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1.0  INTRODUCTION 


Frank  J.  Harris 
U.S.  Army  Research  Institute 


To  conduct  all  military  training  using  the  entire  system  that  an 
individual  will  ultimately  be  involved  with  in  his  operational  mission 
is  logistically  impossible,  cost-wise  prohibitive,  and  probably  not  the 
most  efficient  way  to  train  anyway. 

Designers  of  training  systems  are  faced  with  the  problem  of  deter¬ 
mining  how  much  of  the  physical  aspects  of  a  system  must  be  faithfully 
reproduced  in  order  to  elicit  sufficient  psychological  realism  that  the 
training  is  meaningful  and  will  indeed  transfer  to  the  ultimate  opera¬ 
tional  mission. 

Put  another  way,  given  that  it  is  impossible  and  probably  even  un¬ 
desirable  to  reproduce  the  complete  operational  environment  in  a  train¬ 
ing  setting,  how  does  the  training  designer  find  that  minimal  or  optional 
level  of  physical  fidelity  to  achieve  an  acceptable  level  of  psychologi¬ 
cal  fidelity  sufficient  to  elicit  the  behaviors  or  performance  levels 
specified  as  training  goals  or  objectives? 

The  question  of  course  arises  as  to  how  the  training  system  designer 
decides  when  psychological  fidelity  has  been  achieved.  What  sort  of 
analyses  of  psychological  factors  are  required  so  that  the  training  sys¬ 
tem  designer  knows  when  he  or  she  has  enough  physical  fidelity  to  elicit 
the  behaviors  critical  for  training?  And  when  these  behaviors  are  elic¬ 
ited,  how  does  the  training  system  designer  know  that  there  is  indeed 
psychological  equivalence  between  the  simulated  and  the  operational 
environment? 

We  have  asked  paper  presenters  to  describe  how  psychological  fidel¬ 
ity  requirements  and/or  behavioral  contingencies  were  isolated  in  the 
operational  environment  for  simulation  in  the  training  environment;  what 
training  goals  or  performance  objectives  were  established  prior  to  or 
during  simulation  design;  how  performance  evaluation  or  assessment  was 
handled  in  the  context  of  simulation;  and  what  sort  of  cost/benefit  trade¬ 
offs  seem  to  exist  in  focusing  on  achieving  psychological  fidelity  rather 
than  full-scale  simulation  for  the  work  environment  with  which  they  have 
been  involved. 

Our  paper  presenters  come  from  the  Air  Force,  industry,  and  the 
Army,  and  their  simulation  interests  range  from  individual  to  collective 
training  in  both  machine-ascendant  and  what  we  might  call  "man-ascendant" 
systems,  in  which  system  performance  is  a  function  of  man-man  as  well  as 
man -machine  interactions. 
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2.0  THE  CONCEPT  OF  PERFORMANCE  EQUIVALENCE  IN  TRAINING  SYSTEMS 


W.  G.  Matheny 
Life  Sciences,  Inc. 


Our  topic  of  concern  is  psychological  fidelity  in  simulated  work 
environments.  Most  often  when  we  concern  ourselves  with  psychological 
fidelity  we  are  working  in  the  area  of  training.  More  specifically, 
we  are  generally  concerned  that  a  training  device  bear  some  fidelity 
relationship  to  some  real  target  system.  However,  we  may  be  concerned 
with  the  psychological  fidelity  between  two  situations  for  purposes  of 
selection  and  classification;  system  design  or  in  studying  the  effect 
of  such  operator  variables  as  age,  sex,  fatigue,  stress,  level  of  ex¬ 
perience,  or  other  biographical,  physiological  or  psychological  descrip¬ 
tors  using  a  simulated  system. 

However,  usually  we  use  the  term  psychological  fidelity  in  the  con¬ 
text  of  training  and  our  concern  is  with  bringing  about  a  psychological 
realism  rather  than  striving  for  physical  realism  between  the  two  systems. 
I  will  be  speaking  primarily  about  fidelity  for  training  in  this  presen¬ 
tation,  although  the  fact  that  we  may  be  interested  in  psychological 
fidelity  for  a  number  of  other  purposes  should  be  borne  in  mind. 

When  we  address  ourselves  to  the  question  of  the  psychological  fi¬ 
delity  of  two  systems,  the  implication  is  that  we  are  looking  for  some 
way  in  which  the  two  systems  are  alike  and  that  this  "alikeness"  is  some 
characteristic  of  a  human  within  the  system.  The  term  "fidelity"  as 
meaning  the  accuracy  of  a  description,  a  translation  or  a  reproduction 
(a  la  Webster)  is  quite  unambiguous  and  useful.  The  term  "psychological" 
is  the  one  which  gives  us  the  problem  and  which  must  be  clarified  and 
defined. 

By  the  term  psychological  we  mean  some  behavior  on  the  part  of  the 
operator  of  the  system  which  we  may  observe  and  measure  and  relate  to 
something  meaningful.  As  a  first  step  toward  a  definition,  let  us  say 
that  there  is  some  psychological  fidelity  relationship  holding  between 
two  systems  when  the  two  elicit  the  same  behavior  on  the  part  of  the 
human  operator.  Let  us  then  talk  about  behavioral  fidelity  rather  than 
psychological  so  that  we  are  steered  toward  looking  for  and  operationally 
defining  some  observable  behavior.  There  is  then  behavioral  fidelity  be¬ 
tween  the  two  systems  or  situations  when  they  both,  under  the  same  cir¬ 
cumstances,  elicit  the  same  operator  behavior. 

Since  we  are  trying  to  move  toward  designing  systems  in  terms  of 
their  behavioral  rather  than  physical  fidelity,  we  may  take  our  cue 
from  psychophysics  and  seek  to  define  the  parameters  of  interest  on  a 
behavioral  continuum  rather  than  the  physical.  However,  practically 
speaking,  for  most  training  systems,  we  must  stay  rooted  in  the  physi¬ 
cally  measurable  in  order  that  we  may  design  and  build,  although  our 
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concern  is  with  the  behavioral  concomitants  of  these  physical  param¬ 
eters.  This  rooting  in  the  physical  world  is  necessary  in  order  to 
translate  behavioral  fidelity  into  physical  reality.  The  important 
point  is  that  the  behavioral  requirements  dictate  the  physical  charac¬ 
teristics  rather  than  some  perceived  physical  fidelity  dictating  be¬ 
havior.  Parenthetically,  the  definition  of  the  physical  continuum 
underlying  the  behavioral  is  not  always  easy  since  they  must  be  stated 
in  terms  meaningful  to  the  design  engineer  and  allow  him  flexibility 
in  exercising  his  ingenuity  in  bringing  into  being  hardware  which  will 
satisfy  the  behavioral  requirements.  This  definition  of  the  physical 
stimulus  in  terms  meaningful  to  the  engineer  and  which  allows  him  to 
exercise  his  art  in  bringing  the  stimulus  into  functional  hardware  is 
the  key  to  the  symbiotic  relationship  between  psychologist  and  engineer 
spoken  of  by  Col.  Dan  Fulgham  in  his  recent  paper  before  the  Royal  Aero¬ 
nautical  Society  Flight  Simulation  Group  (7). 

We  have  said  that  psychological  fidelity  implies  some  sort  of  be¬ 
havior  exhibited  as  some  measurable  performance.  We  may  then  define 
the  performance  we  are  interested  in  as  being  the  metric  for  measuring 
degree  of  fidelity.  Again  taking  our  cue  from  psychophysics,  while  we 
stay  rooted  in  the  physical  measurement  of  certain  parameters,  our  con¬ 
tinue  of  interest  are  those  behavioral  concomitants  of  variations  in 
the  physical  since  it  is  human  operator  behavior  which  we  want  to  under¬ 
stand  and  to  be  able  to  predict.  From  this  premise  I  want  to  examine 
psychological  fidelity  in  the  context  of  a  simulator  of  am  aircraft. 
Certain  generalizations  might  be  drawn  from  this  specific  case  which  may 
be  reasonable  and  useful. 

Taking  our  cue  again  from  psychophysics,  the  problem  that  faces  us 
is  one  of  systematically  varying  the  physical  environment  and  noting  its 
effect  upon  some  behavior.  Unfortunately,  am  aircraft  presents  a  tre¬ 
mendously  complex  set  of  stimuli  to  the  operator  so  we  are  faced  with 
the  question  of  what  realities  to  attend  to  amd  what  stimuli  to  vary. 

We  have  in  fact  turned,  in  the  past,  to  psychophysical  studies  in  seek¬ 
ing  am  answer  to  the  psychological  fidelity  of  simulation  and  have  re¬ 
searched  such  questions  as  "what  is  the  threshold  for  perceived  motion 
in  either  vestibular  or  the  total  body  movement  sense."  We  have  tended 
to  reject  these  findings  for  the  most  part  on  the  basis  that  the  isola¬ 
tion  and  variation  of  single  stimuli  within  the  total  complex  of  what 
is  going  on  in  am  aircraft  is  unrealistic  and  our  object  representation 
is  not  valid. 

Our  problem  then  is  to  identify  some  behavior  which  is  reflective 
of  the  total  complex  of  the  stimuli  in  the  aircraft  control  situation 
which  is  measurable  amd  meaningful.  We  then  must  determine  what  we  can 
point  to  in  the  real  physical  world  which  made  the  behavior  happen. 
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One  of  the  persistent  problems  in  simulator  fidelity  is  that  of 
reproducing  the  requirements  for  closed-loop  control  of  the  system. 
Closed-loop  control  or  tracking  is  a  major  part  of  the  task  of  the  op¬ 
erator  of  most  mem/machine  systems,  particularly  aircraft,  and  the 
problem  of  getting  the  simulator  to  "fly  like"  or  "handle  like"  the 
target  system  is  a  major  concern.  Let  us  take  this  closed-loop  control 
aspect  of  the  task  as  one  for  which  we  would  like  to  establish  a  measure 
of  behavior  and  from  there  work  toward  how  we  might  define  behavioral, 
ergo,  psychological  fidelity. 

Let  us  make  clear  that  it  is  operator  behavior  we  are  interested 
in  as  opposed  to  total  system  behavior.  In  an  aircraft,  for  example, 
we  are  interested  in  some  measure  of  the  operator's  output  rather  than 
such  measures  of  total  system  output  as  attitude,  airspeed,  and  the 
like.  Our  interest  in  the  operator's  behavior  centers  on  his  input  to 
the  controls  since  this  is  the  medium  by  which  he  communicates  commands 
to  the  machine.  Without  attempting  to  identify  at  this  time  the  whole 
myriad  of  stimuli  presented  to  the  operator,  let  us  assume  that  the 
aircraft  system  is  so  designed  that,  if  we  wish  it  to  act  in  a  certain 
way,  particular  inputs  must  be  made  into  the  controls.  Any  controller, 
mechanical  or  human,  in  order  to  produce  a  particular  machine  system 
output  must  make  control  inputs  of  a  given  nature.  The  human  operator 
"talks"  to  the  machine  through  the  controls  and  we  may  focus  upon  this 
as  a  means  of  measuring  the  behavioral  requirements  of  the  machine  be¬ 
havior.  We  may  even  vary  certain  physical  dimensions  or  characteristics 
of  the  system  to  determine  whether  this  behavior  changes  and  then  estab¬ 
lish  a  psychophysical  relationship,  if  you  will,  between  physical  and 
behavioral  continua  in  the  basic  psychophysical  sense. 

The  central  question  is  how  do  we  measure  this  operator  behavior. 

In  the  aircraft  industry  we  have  been  measuring  it  for  years  through 
pilot  opinion,  that  is,  a  verbal  expression  of  the  goodness  or  badness 
of  the  handling  qualities  of  the  system.  With  flight  simulators  the 
operator  has  been  asked  to  compare  his  behavior  in  controlling  the  air¬ 
craft  to  the  behavior  he  exhibits  in  controlling  the  simulator  and  then 
to  express  his  opinion  as  to  the  degree  to  which  they  are  alike.  In 
order  for  the  measurement  to  be  as  reliable  and  discriminating  as  possi¬ 
ble,  we  have  selected  the  most  experienced  operator  of  the  system  pos¬ 
sible  and  supplied  him  with  rating  scales  to  assist  him  in  objectifying 
his  opinions.  This  behavioral  measure  has  not  been  entirely  satisfac¬ 
tory.  However,  it  does  look  at  the  system  from  the  behavioral  point  of 
view. 

Another  method  by  which  we  have  asked  whether  two  systems  are  alike 
in  behavioral  terms  is  to  train  individuals  in  a  system  and  then  trans¬ 
fer  them  to  a  target  system  and  to  measure  how  long  it  takes  him  to 
adapt  to  the  new  system.  This  method  seems  to  me  to  be  really  answer¬ 
ing  a  different  question.  In  it  we  are  asking  what  skills  and  knowl¬ 
edges  gained  prior  to  the  observation  of  the  behavior  are  utilized  in 
the  target  system.  More  importantly,  we  are  asking  how  long  does  it 


tako  the  operator  to  learn  to  adapt  these  skills  and  knowledges  to  the 
target  system.  Tho  measures  wc  take  in  such  a  situation  have  tradi¬ 
tionally  been  those  of  tho  total  system  output  such  as  the  aircraft 
altitude,  airspeed,  and  so  forth  that  we  montioned  earlier.  It  does 
not  scorn  to  me,  however,  that  this  is  a  particularly  suitable  method 
for  establishing  tho  behavioral  fidolity  of  one  system  versus  another. 

It  is  indeed  tho  only  way  we  havo  of  establishing  tho  effectiveness  of 
a  given  system  fot  training  in  the  target  system  for  the  particular 
level  of  experience  of  the  individuals  in  which  wo  are  interested. 
Generally,  in  transfer  of  training  experiments  wo  are  interested  in 
the  effectiveness  of  the  total  complex  of  tho  training  tool  and  methods 
for  reducinq  training  time  in  the  target  system. 

The  fidelity  quostion  is  only  one  part  of  the  total  question  being 
asked  with  rospect  to  tho  effectiveness  of  a  training  system.  In  ask¬ 
ing  the  fidelity  quostion  we  are  in  essence  asking,  is  there  some  way 
in  which  wo  can  come  to  some  determination  of  tho  faithfulness  with 
which  the  simulated  situation  represents  the  target  situation.  In  this 
wo  seem  all  to  be  tacitly  assuming  that  if  wo  can  establish  a  behavioral 
equivalence  between  our  training  situation  and  the  target  situation  that 
we  can  substitute  training  in  tho  simulated  situation  for  that  in  the 
target  situation  with  some  degree  of  confidence.  Wo  somohow  believe 
that  if  a  simulated  situation  or  system  requiros  the  same  behavior  an 
the  targot  system,  then  training  in  that  system  should  be  beneficial 
and  effective.  At  the  same  time  wo  somehow  believe  that  the  physical 
realism  and  fidelity  which  we  are  now  building  into  many  systems  goes 
boyond  what  is  necessary  to  bring  about  behavioral  fidelity. 

We  are  continually  faced  with  the  very  real  question  in  building 
training  systems  that  we  must  design  and  build  some  hardware  components 
of  that  system  which  provide  practice  for  the  student.  Whether  the 
training  system  within  which  this  hardware  is  embedded  is  efficient 
and  effective  or  not  depends  upon  many  other  variables  apart  from  tho 
fidolity  of  the  training  hardwaro.  Thus,  a  design  engineer,  for  ex¬ 
ample,  is  of ton  asked  to  build  such  a  piece  of  hardwaro  with  very  little 
guidance  as  to  its  training  requirements  other  than  it  is  for  a  particu¬ 
lar  tarqot  system.  On  tho  other  hand,  he  may  receive  very  explicit  and 
detailed  guidance  and  specification  with  respect  to  tho  certain  aspects 
of  the  hardware  involved.  Generally,  he  has  little  guidance  with  ro¬ 
spect  to  the  way  in  which  he  must  design  in  order  to  create  particular 
specif ied  behaviors  on  the  part  of  the  operator.  When  he  is  asked  to 
design  in  accordance  with  Military  Standard  1472,  he  has  some  assurance 
that  his  anthropometric  and  knobs  and  dials  design  will  result  in  cer¬ 
tain  behavior.  However,  when  he  tries  to  desiqn  an  aircraft  simulator 
so  that  it  will  produce  the  appropriate  control  behavior,  he  has  prac¬ 
tically  no  knowledge  of  tho  limits  to  place  on  his  physical  design. 

This  is  particularly  true  for  tho  area  of  aerodynamic  equations,  visual 
scene,  and  motion  characteristics. 


6 


How  might  we  attack  this  whole  problem,  then,  of  providing  infor¬ 
mation  to  a  designer  such  that  he  could  use  his  ingenuity  to  provide 
effectively  the  necessary  and  sufficient  hardware  to  bring  about  the 
desired  behavior,  i.e.,  the  behavior  that  would  be  exhibited  in  the  tar¬ 
get  system?  First,  we  must  define  the  behavior  and  how  we  might  measure 
it. 

As  shown  in  Figure  2-1,  we  can  take  as  the  behavior  of  interest 
that  of  the  output  of  the  operator  into  the  controls  of  the  system  since 
this  is  a  manner  in  which  he  communicates  to  the  system  and  which  we 
will  assume  has  a  characteristic  signature  in  order  to  bring  about  par¬ 
ticular  outputs  of  the  system.  The  control  engineer,  when  designing  an 
electro-mechanical  control  system,  seeks  to  "match"  the  system's  dynamic 
characteristics  in  order  to  model  a  control  device.  Methods  for  describ¬ 
ing  the  way  in  which  the  human  controller  "matches"  the  system's  dynamics 
and  exercises  control  are  not  fully  determined.  The  present  state  of 
this  area  of  investigation  has  been  summarized  very  professionally  by 
Pew  (14).  We  will  come  to  a  more  specific  operational  definition  of  the 
operator  behavior  of  interest  to  us  after  a  bit  more  background 
discussion. 
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Figure  2-1.  Performance  measurement  points  (MP)  in  the 
man/machine  system. 

Craik  in  1946  (3);  Benepe,  Narasimhan,  and  Ellson  in  1954  (1);  and 
Fitts,  Bennett,  and  Bahrick  in  1956  (5)  summarized  the  control  input  sig¬ 
nature  of  the  operator  through  power  density  spectrum  analysis  and  auto- 
correlational  techniques.  While  Fitts  was  interested  in  understanding 
the  nature  of  skilled  motor  behavior,  Benepe  et  al.  were  more  interested 
in  a  description  of  the  operator  for  system  design  purposes.  Suggestions 
from  these  studies  formed  the  basis  for  my  early  interest  in  this  area. 
For  example,  Fitts  indicated  that  the  cross-correlation  function  provided 
a  measure  of  the  average  lead  or  lag  of  the  response  relative  to  the 
stimulus  in  the  tracking  situation,  that  is,  the  measure  was  sensitive 
to  physically  measurable  changes  in  the  system.  Benepe  et  al.  showed 


chat  while  the  power  density  spectral  composition  varied  with  practice, 
after  attaining  skill  in  the  system  the  operator  exhibited  stationary 
control  behavior.  This  conclusion  appears  at  our  present  state  of 
knowledge  to  be  an  oversimplification.  However,  in  Benepe's  experiment 
it  is  important  to  note  that  stationary  behavior  occurred  in  that  situ¬ 
ation  in  which  the  visual  signal  was  accompanied  by  a  synchronized  audi¬ 
tory  signal.  He  suggested  that  we  cannot  consider  the  human  operator 
as  an  isolated,  single  channel  component  of  a  system  since,  when  the 
operator  was  obtaining  information  from  moro  than  one  channel,  he  was 
able  to  maintain  highly  consistent  behavior  as  indicated  by  the  distri¬ 
bution  of  power  in  his  control  output. 

My  own  interest  in  the  fidelity  of  simulation  problem  originated 
as  a  result  of  complaints  I  received  from  pilot  traineos  during  World 
War  II  directed  at  the  old  "blue  box"  Link  instrument  trainers.  This 
interest  came  to  focus  in  1950  when  Williams  and  I  carried  out  a  survey 
in  which  we  asked  the  question  "what  is  the  necessary  and  sufficient 
hardware  to  be  built  inko  a  trainer  in  order  to  provide  effective  train¬ 
ing  for  a  particular  system."  The  answer  at  that  time  was  "we  don't 
know  and  it  costs  but  little  more  to  add  the  hardware  for  any  charac¬ 
teristic,  so  why  worry."  That  was  our  answer.  The  same  question  has 
been  raised  over  the  last  27  years  with  essentially  the  same  answer, 
and  we  are  addressing  it  here  again  today. 

In  the  early  fifties  we  attempted  to  get  some  handle  on  the  prob¬ 
lem  in  both  the  laboratory  and  in  very  simplified  maneuvers  in  the  simu¬ 
lator  using  transfer  of  training  as  the  evaluative  measure.  In  labora¬ 
tory  experiments  carried  out  by  Muckier  and  Matheny  (13)  the  amount  of 
friction  and  direction  of  movement  of  the  control  forces  in  a  tracking 
task  were  varied  to  determine  their  effect  upon  transfer  to  a  criterion 
task.  It  was  found  for  original  learning  and  for  transfer  that  neither 
differing  amounts  of  control  friction  nor  direction  of  movement  of  the 
control  had  a  differential  effect  upon  transfer  when  these  variables  were 
studied  as  independent  variables.  The  interesting  effect,  however,  came 
about  when  both  control  force  and  direction  of  movement  were  changod  and 
negative  transfer  of  training  occurred.  This  has  significant  implica¬ 
tions  for  the  study  of  behavioral  fidelity  using  minute  and  isolated  as¬ 
pects  of  the  task. 

We  later  studied  the  effect  of  varying  control  forces  on  the  trans¬ 
fer  of  training  between  the  simulator  and  the  AT-6  aircraft.  The  find¬ 
ings  showed  that  both  for  time  to  reach  asymptote  in  original  learning 
and  for  transfer  of  training  no  differential  effects  of  control  force 
were  found  for  the  control  forces  studied. 

Later  in  1957  and  1958  in  the  conduct  of  the  Army/Navy  instrumen¬ 
tation  program,  the  Army  and  the  Navy  supported  the  construction  of  a 
simulator  at  Bell  Helicopter  for  use  in  instrumentation  studies.  We 
were  not  only  interested  in  whether  or  not  different  designs  brought 
about  different  control  behaviors  on  the  part  of  the  operator  but  were 
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also  interested  in  isolating  insofar  as  possible  the  cues  used  by  the 
operator  in  carrying  out  his  task.  We  were,  therefore,  able  to  obtain 
data  on  operator  inputs  performing  the  hovering  maneuver  in  the  simu¬ 
lator  both  with  and  without  motion.  The  autocorrelations  of  the  out¬ 
puts  shown  in  Figure  2-2  represent  in  a  simplified  way  the  essence  of 
the  behavioral  fidelity  approach  for  determining  the  requirements  for 
simulators.  Figure  2-2  illustrates  the  shift  in  operator  control  be¬ 
havior  under  conditions  of  motion  and  no  motion  which  could  be  consid¬ 
ered  a  behavioral  index  of  change  in  the  physical  system.  Under  other 
conditions  of  system  dynamics,  external  forcing  function  or  system  lim¬ 
its  no  such  shift  in  control  behavior  might  be  found. 

Later,  in  1964  and  1965  we  carried  out  a  series  of  experiments 
using  the  UDOFTT  simulator  at  NTDC  in  which  the  question  of  the  fidel¬ 
ity  with  which  the  equations  of  motion  of  a  high  performance  interceptor 
aircraft  need  be  reproduced  in  the  simulator  was  tested.  In  this  series 
of  experiments  both  transfer  of  traininq  and  control  input  data  were 
collected.  Conditions  were  tested  in  which  the  equations  of  motions 
were  implemented  to  the  fullest  extent  possible  or  were  highly  simpli¬ 
fied,  i.e.,  straight  line  approximations  with  minimal  break  point  were 
used.  A  second  variable  of  interest  at  that  time  was  the  program  cycle 
time  for  updating  the  equations  of  motion.  Interestingly,  this  cycle 
time  problem  underscores  the  necessity  for  a  reliable  and  quick  method 
for  determining  the  behavioral  impact  of  hardware  questions.  We  need 
a  method  for  quickly  and  effectively  assessing  whether  or  not  a  problem 
as  seen  by  engineers  really  is  a  behavioral  problem  and  whether  or  not 
a  hardware  innovation,  however  ingenious,  is  really  a  step  forward  in 
terms  of  its  behavioral  impact. 

The  results  of  the  series  of  UDOFTT  experiments  indicated  that 
quite  drastic  simplification  of  the  equations  could  be  instituted  with 
no  change  in  original  learning  or  transfer  and,  more  particularly,  with 
no  change  in  the  required  control  inputs  in  order  to  execute  control  of 
the  system  in  terms  of  the  measures  used  to  summarize  the  control  in¬ 
puts.  As  a  footnote,  very  soon  after  these  experiments  were  completed 
several  quantum  jumps  in  computer  technology  vastly  increased  the  mem¬ 
ory  available,  so  again  we  trod  the  path  of  physical  fidelity  in  imple¬ 
menting  equations  of  motion  rather  than  following  any  indications  of 
the  behavioral  requirements  which  had  been  established,  limited  as  they 
might  be. 

In  1973  and  1974,  in  an  attempt  to  gain  information  with  respect 
to  the  fidelity  requirements  for  motion  in  ground  based  trainers, 

Matheny,  Lowes,  and  Bynum  (10)  carried  out  a  study  using  the  TRADEC 
simulator  at  NTDC.  Here  again  the  essential  question  asked  was  "will 
changes  in  the  motion  cues  provided  to  the  operator  change  his  control 
behavior?"  In  this  study  we  were  also  concerned  with  defining  the 
physical  variable  in  ways  which  were  not  tied  to  specific  pieces  of 
equipment  or  methods  of  implementation.  Rather,  we  wanted  a  general 
description  of  the  motion  variable  which  could  be  implemented  in 
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CYCLIC  PITCH 

MEAN  SQUARE  VALUE 
NO  MOTION  -  183.7 
MOTION  -  93.3 


Cyclic  pitch  output. 


CYCLIC  ROLL 
MEAN  SQUARE  VALUE 
NO  MOTION  -  174.5 
MOTION-  68.8 
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Tim*  Shift  (t) 

Cyclic  roll  output. 

Figure  2-2.  Shifts  In  operator  control  behavior 


different  ways  by  design  engineers  according  to  the  state-of-the-art 
or  the  constraints  of  cost  and  time.  In  these  studies  the  measure  of 
the  operator's  control  behavior  was  a  simple  system  identification 
method  in  which  a  Bode  plot  of  the  transfer  operator's  output  was  gen¬ 
erated  and  the  lower  and  upper  corner  frequencies  were  identified.  As 
in  former  studies  a  shift  in  controller  output  frequencies  under  differ¬ 
ent  conditions  of  the  system  was  evident.  It  is  of  interest  that  this 
behavioral  metric  shows  rather  nicely  a  shift  of  the  operator's  inputs 
to  high  frequencies  of  an  anticipatory  nature  when  motion  cues  were 
present  in  this  rather  high  frequency  response  system. 

Incidentally,  in  the  latter  two  studies,  i.e.,  those  with  the 
UDOFTT  and  those  with  the  TRADEC,  the  pilot  ratings  of  the  systems  were 
obtained.  The  results  were  that  there  was  great  variability  among  raters 
with  little  relationship  between  the  ratings  and  performance. 

Most  recently  an  attempt  was  made  to  obtain  data  in  an  instrumented 
aircraft  which  could  be  used  to  describe  the  operator's  behavior  in  con¬ 
trolling  that  system  under  specified  conditions.  It  was  the  intent  that 
behavioral  indices  might  be  developed  and  used  for  comparing  aircraft 
and  simulator  data  in  terms  of  behavioral  fidelity.  A  word  needs  to  be 
said  here  about  the  level  of  experience  and  expertise  of  the  operator 
of  the  system  when  gathering  data  of  this  nature.  Two  points  need  to 
be  made.  First,  the  operator  of  the  system  needs  to  be  highly  experi¬ 
enced  since  he  is  being  used  as  a  "measuring  instrument”  in  determining 
the  control  behavior  required  to  control  the  system  according  to  certain 
specified  conditions.  Second,  from  a  training  point  of  view,  getting 
information  about  the  control  behavior  of  the  highly  experienced  and 
proficient  pilot  should  be  very  helpful  in  instruction  or  adaptive 
training  situations  since  it  is  this  type  of  behavior  which  one  is  try¬ 
ing  to  instill  through  training  in  the  student. 

In  this  investigation  data  were  collected  in  an  instrumented  T-37 
aircraft  with  experienced  pilots  flying  prescribed  maneuvers  to  produce 
system  outputs  within  certain  prescribed  limits.  In  all,  22  variables 
were  recorded  for  analysis,  five  of  which  were  pilot  output  or  control 
input  variables.  These  were  elevator,  aileron,  rudder,  aileron  trim 
tab,  and  throttle  positions.  The  other  17  variables  were  hypothesized 
to  be  measures  of  inputs  to  the  pilot.  Those  variables  which  could 
serve  as  inputs  to  one  or  more  of  the  pilot's  perception  modalities 
were  such  parameters  as  control  forces,  airspeed,  altitude,  pitch  angle, 
pitch  rate,  and  so  on.  Similar  data  were  scheduled  to  be  collected  in 
a  simulator  of  the  T-37  aircraft;  however,  only  one  pilot  run  was  re¬ 
corded  due  to  a  number  of  administrative  and  technical  problems. 

The  method  of  analysis  here  was  intended  to  be  much  more  compre¬ 
hensive  and  inclusive  of  more  variables  than  any  used  in  previously  men¬ 
tioned  research.  The  analytical  approach  was  to  develop  a  linear,  time 
variable,  pilot  identification  of  the  pilot  flying  the  T-37  in  carrying 
out  the  specified  maneuvers.  A  review  of  the  methods  and  application 


of  system  identification  techniques  can  be  found  in  the  April  1977  is¬ 
sue  of  Naval  Research  Reviews  and  an  introduction  to  then  is  given  in 
Brogan's  (2)  Modern  Control  Theory.  Anyone  seriously  interested  in 
this  approach  to  behavior  identification  should  read  Pew  et  al.'s  (14) 
March  1977  review  and  analysis  of  performance  models.  Due  to  limita¬ 
tions  in  funds  and  time,  only  a  preliminary  analysis  of  the  data  could 
be  made.  The  analysis  indicated  that  the  data  were  adequate  for  use  in 
carrying  out  an  identification  procedure  aimed  at  identifying  parameters 
and  coefficients  describing  the  operator  in  the  aircraft.  The  data 
could  also  be  used  to  compare  these  measures  across  pilots  and  to  as¬ 
sess  their  generality.  More  importantly,  perhaps,  the  data  could  be 
used  to  compare  the  results  obtained  in  the  aircraft  to  those  from  the 
simulator  as  a  test  of  its  behavioral  fidelity  within  the  limits  of 
the  identification  model. . 

In  summary,  the  thesis  presented  here  is  that  we  may  establish  the 
behavioral  fidelity  of  a  simulator  for  the  control  aspects  of  the  task 
through  application  of  system  identification  procedures.  What  is  sug¬ 
gested  is  that  when  we  speak  of  behavioral  fidelity,  we  can,  for  the 
closed-loop  control  task,  look  at  the  operator's  behavior  and  describe 
it  by  system  identification  methods.  This  appears  to  be  a  feasible  and 
potentially  high  payoff  approach  to  determining  whether  two  systems  are 
behaviorally,  i.e.,  psychologically,  equivalent.  If  two  systems  are 
behaviorally  equivalent,  we  might  assume  that  training  in  one  would 
transfer  positively  to  the  other  for  this  aspect  of  the  task — an  as¬ 
sumption  which  could  be  tested  empirically.  An  advantage  of  such  an 
approach  is  that  it  could  be  used  to  investigate  specific  characteris¬ 
tics  of  simulators  both  with  regard  to  their  similarity  to  the  systems 
they  simulate  and  the  degree  to  which  physical  changes  in  them  are  re¬ 
flected  in  changes  in  the  operator's  behavior. 

We  are  very  little  closer  to  answers  to  the  behavioral  similarity 
question  today  than  we  were  25  years  ago.  It  is  not  likely  that  we 
will  get  there  through  transfer  of  training  experiments  since  these  are 
aimed  at  a  quite  different  question.  Transfer  experiments  include  vari¬ 
ables  such  as  the  use  to  which  the  simulator  is  put,  methods  of  instruc¬ 
tion,  and  so  forth  and  the  operator's  behavior  is  continually  changing 
due  to  learning.  In  such  an  experiment  the  positive  effects  of  a  par¬ 
ticular  simulation  hardware  could  well  be  offset  by  the  negative  trans¬ 
fer  effects  of  method  of  instruction  or  other  variables.  Thus,  in  such 
experiments  we  have  a  definitive  test  of  the  transfer  effectiveness  of 
a  particular  training  system  used  in  a  particular  way  but  by  its  very 
nature  does  not  give  us  definitive  information  about  the  fidelity  of 
simulation  questions.  It  seems  to  me  we  must  attack  this  problem  in 
a  different  way. 

Paul  Fitts,  in  his  book  Skilled  Performance,  seems  to  me  to  have 
provided  a  clue  to  this  whole  approach  when  he  wrote,  "in  many  of  the 
tasks  performed  in  industry  and  in  military  service  an  important  factor 
is  the  dynamic  characteristics  of  the  physical  systems  such  as  submarines , 
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automobiles,  or  other  machines  that  are  operated  by  man.  The  1  «.\g a  and 
oscillations  characteristic  of  such  systems  determine  part  of  what  a 
man  must  learn  in  controlling  them." 

To  me  the  essential  ingredients  for  gaining  some  handhold  on  the 
problem  are  those t 

1.  The  test  of  the  system  identification  procedure  for  describ¬ 
ing  the  behavior  of  the  operator  as  he  makes  control  inputs 
into  tho  systems  of  intorest. 

2.  The  definition  of  the  physical  variables  going  into  training 
simulator  systems,  and  in  terms  which  are  generalizable  and 
are  not  situation  or  hardware  spocific. 

3.  The  accumulation  of  identification  data  for  human  operator 
behavior  when  controlling  systems  across  representative  sets 
of  conditions. 

4.  In  order  to  carry  out  a  comprehensive  program  of  research  in 
this  area,  the  availability  of  an  instrumented  system  and  a 
simulator  of  that  system  whose  characteristics  can  bo  varied 
over  a  wide  range,  and  in  which  important  external  variables 
can  be  closely  controlled  and  made  representative  of  those 
in  the  target  system. 

With  this  approach  and  these  tools  I  believe  the  psychologist  and  the 
engineer  could  work  in  a  symbiotic  relationship  providing  answers  as  to 
what  brings  about  psychological — or  more  precisely — behavioral  fidelity. 
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Introduction 


Several  reviews  of  the  team  training  area  have  appeared  in  the  re¬ 
cent  literature  (e.g.,  Hall  &  Rizzo,  1975;  Meister,  1976;  Faust,  1976; 
Wagner,  Hibbits,  Rosenblatt,  &  Schulz,  1977;  Collins,  1977;  Kribs,  Thur¬ 
mond,  £  Mark,  1977).  Each  of  the  reviews  has  included  an  examination 
of  current  team  training  technology  and  an  identification  of  the  major 
shortcomings  of  that  technology.  One  shortcoming  consistently  cited  in 
these  reviews  is  the  absence  of  an  adequate  task  analytic  or  Instruc¬ 
tional  Systems  Development  (ISD)  technique  for  identification  of  team, 
as  opposed  to  individual  training  requirements.  Absence  of  such  a  tech¬ 
nique  greatly  impairs  the  capability  to  optimally  structure  team  train¬ 
ing  programs  and  design  effective  team  training  devices. 

The  purpose  of  this  paper  is  to  describe  and  discuss  a  task  ana¬ 
lytic  technique  that  has  been  successfully  used  in  a  number  of  applica¬ 
tions,  including  design  of  a  team  training  device  for  members  of  the 
fire  control  team  of  the  AC-130E  Gunship.  The  technique  was  developed 
to  overcome  two  major  weaknesses  of  traditional  ISD  processes:  (1)  the 
lack  of  sufficient  specificity  for  actual  design  of  training  devices; 
and  (2)  the  lack  of  an  adequate  means  to  address  design  of  a  device  for 
team  or  crew  coordination  training.  The  intent  of  the  device  design 
technique  is  to  provide  only  the  levels  of  fidelity  that  are  necessary 
to  accomplish  specific  training  objectives.  The  technique  is  based 
upon  careful  task  analyses  and  represents  a  modification  and  extension 
of  traditional  ISD  techniques  (Cream,  Eggemeier,  £  Klein,  1978) .  This 
paper  provides  a  general  description  of  the  technique,  a  discussion  of 
the  crew  coordination  aspects  of  the  technique,  and  the  results  of  an 
evaluation  of  the  Gunship  team  trainer  which  was  developed  using  the 
technique.  Finally,  the  applicability  of  the  technique  to  "man- 
ascendant"  and  "machine-ascendant"  systems  is  discussed. 


Training  Device  Design  Technique 

The  training  device  design  technique  is  based  upon  the  use  of  be¬ 
havioral  data  in  a  development  process  which  involves  the  intended  users 
of  the  training  device,  training  psychologists,  and  simulation  engineers. 
A  basic  objective  of  the  technique  is  to  provide  a  description  of  the 
training  requirements  that  are  to  be  accomplished  in  the  training  device. 
Training  requirements  are  expressed  in  behavioral  terms.  These  require¬ 
ments  are  eventually  translated  into  training  device  requirements.  The 


user  serves  as  a  subject-matter  expert  in  identifying  the  initial  set 
of  training  requirements.  The  user  also  participates  in  the  iterative 
process  which  is  involved  in  translating  training  requirements  into 
device  requirements.  The  training  psychologist  is  responsible  for  de¬ 
veloping  and  coordinating  inputs  from  the  user.  The  psychologist  also 
serves  as  the  interface  between  the  user  and  simulation  engineer.  The 
engineer  is  responsible  for  implementing  the  training  requirements  and 
producing  a  design  specification  capable  of  satisfying  the  requirements. 

During  initial  stages  of  the  device  design  process,  a  complete 
description  of  stimulus-response  conditions  for  each  control  and  dis¬ 
play  within  the  subject  system  is  developed.  This  first  analysis  in¬ 
cludes  a  listing  of  all  tasks  and  subtasks,  and  an  indication  of  their 
sequencing.  It  also  includes  a  complete  description  of  the  initiating 
and  terminating  conditions,  and  the  specific  actions  required  in  each 
circumstance. 

When  the  behavioral  task  listing  has  been  completed,  the  specific 
capabilities  that  will  be  included  in  the  device  itself  are  determined. 
The  major  decisions  in  this  process  are  selection  of  the  subset  of  tasks 
that  will  actually  be  trained  in  the  device  and  determination  of  the 
degree  of  fidelity  necessary  to  train  each  of  the  tasks  in  the  subset. 

Selection  of  tasks  and  determination  of  necessary  fidelity  is  an 
iterative  process  which  involves  a  more  detailed  analysis  of  each  task. 
Initially,  the  user  ranks  each  task  on  three  dimensions:  criticality 
(C)  ,  difficulty  (D)  ,  and  frequency  (F)  of  performance.  Tasks  with  high 
C/D/F  ratings  are  identified  for  inclusion  in  the  device,  while  tasks 
with  low  ratings  will  be  excluded. 

The  next  iteration  consists  of  further  examination  of  tasks  iden¬ 
tified  for  inclusion  in  the  trainer.  The  purpose  is  to  determine  proper 
levels  of  fidelity  to  support  training  of  the  selected  subset.  During 
this  iteration,  the  user  provides  information  to  identify  essential 
stimulus  and  response  components  of  the  various  controls  and  displays. 
For  example,  orwts  console  of  the  AC-130E  Gunship  trainer  was  a  Forward 
Looking  Infrared  ^FLIR)  station.  The  FLIR  operator  in  the  Gunship  is 
responsible  for  acquiring  and  detecting  targets  using  infrared  imagery. 
In  design  of  the  FLIR  station,  there  were  26  tasks  which,  on  the  basis 
of  C/F/D  ratings,  had  been  identified  for  inclusion  in  the  trainer.  It 
was  necessary  to  define  the  degree  of  fidelity  required  to  support  the 
26  required  tasks.  The  visual  display  variables  that  were  necessary  for 
training  were  identified.  For  example,  the  users  indicated  that  the 
FLIR  visual  stimulus  used  in  target  acquisition  was  a  specific  reflec¬ 
tance  pattern  embedded  in  a  heterogeneous  display  pattern.  It  was  also 
determined  that  this  pattern  was  affected  by  a  number  of  variables  in¬ 
cluding  weather,  time  of  day,  aircraft  speed,  approach  angle,  and  atti¬ 
tude.  The  design  team  focused  on  the  implications  of  not  providing 
each  of  these  variables,  and  determined  that  adequate  training  could 
be  provided  by  simulating  a  limited  number  of  representative  cases. 


Only  a  limited  set  of  weather  conditions  and  aircraft  speeds  was  re¬ 
quired.  It  was  also  determined  that  a  representation  of  the  target 
would  be  satisfactory  and  that  all  target  contextual  effects  did  not 
need  to  be  duplicated. 

Thus,  during  the  iterative  procedure,  the  critical  tasks  to  be 
trained  in  the  device  were  identified  and  the  minimum  level  of  stimulus 
fidelity  in  controls  and  displays  necessary  to  support  training  of 
those  tasks  was  determined. 


Team  and  Crew  Coordination  Applications 

The  Gunship  trainer  was  designed  to  permit  both  crew  coordination 
training  and  individual  training.  It  was  therefore  necessary  to  devel¬ 
op  crew  coordination  training  requirements  in  addition  to  training  re¬ 
quirements  that  focused  on  the  individual  team  members  (Cream,  1974)  . 
Training  consoles  for  four  crew  positions  were  developed:  the  FLIR  sen¬ 
sor  operator;  and  the  Low  Light  Level  TV  (LLLTV)  sensor  operator,  who 
detects  targets  using  electronic  imagery;  the  Black  Crow  Electric  War¬ 
fare  (BC/EWO)  sensor  operator,  who  protects  the  aircraft  from  enemy 
radar;  and  the  Fire  Control  Officer  (FCO)  ,  who  is  responsible  for  co¬ 
ordinating  inputs  from  the  sensor  operators.  Crew  members  communicate 
with  each  other  verbally  and  also  respond  to  system  displays  that  pro¬ 
vide  information  from  other  crew  positions. 

Because  of  deficiencies  of  traditional  task  analysis  techniques 
for  dealing  with  team/crew  coordination  activities,  an  extension  of  the 
previously  described  methods  used  for  individual  task  requirements  was 
developed  (Cream,  1974;  Cream  et  al.,  1978).  This  expanded  analysis 
was  based  on  the  results  of  the  original  C/F/D  task  description  which 
had  shown  that  over  one-half  of  the  700  tasks  identified  for  training 
involved  interaction  of  either  system,  individual  crew  members,  or 
both.  Another  crew  member  was  therefore  involved  in  either  the  initi¬ 
ation,  action,  or  conclusion  of  the  majority  of  individual  tasks.  Be¬ 
cause  of  the  high  degree  of  crew  coordination,  a  method  was  developed 
which  separated  the  total  aircraft  mission  into  logical  segments.  With¬ 
in  each  segment,  the  specific  tasks  performed  by  each  of  the  fire  con¬ 
trol  team  members  were  listed.  This  specification  included,  among 
other  items,  an  approximation  of  the  time  required  to  perform  each  task 
by  each  crew  member.  The  purpose  of  this  analysis  was  to:  (1)  describe 
the  interactions  that  occurred  among  team  members  when  performing  coor¬ 
dinated  actions;  and  (2)  to  gain  an  appreciation  of  the  approximate 
task  loading,  not  only  for  the  duties  of  each  team  member  at  each  flight 
segment,  but  also  for  actions  that  resulted  from  the  combination  of 
single  tasks.  Not  surprisingly,  it  was  discovered  that  some  tasks 
which  had  originally  been  rated  as  low  in  difficulty  or  criticality 
when  performed  by  a  single  crew  member  became  more  critical  or  more 
complex  when  performed  within  the  mission  context.  For  example,  when 
a  target  is  located  by  a  sensor  operator,  this  data  is  entered  into 
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the  fire  control  system.  However,  before  the  Aircraft  Commander  can 
fire  the  weapons,  each  one  of  the  team  members  has  to  give  "permission" 
through  both  electronic  and  verbal  communication.  This  apparently  sim¬ 
ple  act,  if  disrupted  or  performed  in  an  inappropriate  sequence,  could 
cause  the  aircraft  weapons  system  to  malfunction.  The  number  of  sepa¬ 
rate  actions  that  each  crew  member  was  required  to  perform  prior  to 
"consent"  varied  according  to  the  sensor  system,  target  type,  aircraft, 
and  electronic  conditions.  Until  the  crew  coordination  task/timeline 
analysis  was  performed,  this  series  of  individually  simple  but  collec¬ 
tively  complex  interactions  was  not  apparent.  There  were  important  im¬ 
plications  for  design  of  the  trainer.  Provisions  had  to  be  made  to 
permit  these  situational,  context  rich  events  to  occur  at  both  the  ap¬ 
propriate  time  intervals  and  with  the  correct  system  displays.  The  re¬ 
quirement  to  provide  realistic  time  intervals  led  to  a  further  problem, 
i.e.,  the  question  of  appropriate  complexity  and  system  fidelity  versus 
instructional  capability.  Specifically,  it  was  necessary  to  consider 
the  degree  to  which  the  trainer  should  permit  pacing  of  the  crew  per¬ 
formance  duties  according  to  the  skill  and  proficiency  of  individual 
team  members  and  the  possible  limitations  that  such  pacing  would  have 
on  development  of  appropriate  experiential  team  skills  due  to  trainer 
processing  delays.  A  close  examination  of  these  factors  and  the  actual 
tasks  that  were  time  critical  or  system  dependent  caused  a  modification 
of  some  design  concepts  that  had  been  based  originally  only  on  the  indi¬ 
vidual  crew  tasks.  For  example,  modifications  were  necessary  at  the 
fire  control  officer  (FCO)  position.  The  actual  FCO  equipment  made  use 
of  a  computer  to  analyze  target  information.  The  trainer  did  not  use  a 
computer  but  instead  required  the  instructor  to  manipulate  an  electronic 
desk  calculator  to  compute  and  insert  the  appropriate  system  target  in¬ 
formation  at  the  student  FCO  station.  The  decision  to  replace  the  com¬ 
puter  was  based  on  the  original  C/F/D  analysis  which  indicated  that  this 
approach  would  permit  training  of  critical  skills  with  a  reduction  in 
overall  trainer  cost.  It  was  apparent  that  this  approach  would  cause 
non-real  time  solutions  for  a  number  of  the  more  complex  fire  control 
problems.  However,  it  was  believed  that  such  time  differences  would  not 
cause  training  problems  if  accompanied  by  specific  modification  of  rele¬ 
vant  portions  of  the  training  scenarios.  These  modifications  were  simply 
to  require  the  other  trainees  to  perform  additional  related  tasks  during 
the  periods  when  these  processing  time  delays  would  be  most  obvious.  In 
practice,  of  course,  this  arrangement  was  not  as  satisfactory  as  expected. 
The  processing  time  delays  were  of  no  consequence  when  the  instructor  and 
the  FCO  student  were  operating  in  an  individual  mode.  However,  when  the 
FCO  student  was  required  to  participate  as  a  member  of  the  team,  these 
delays  resulted  in  problems  for  certain  highly  time-dependent  tasks.  For 
example,  the  students  at  the  other  three  stations  had  been  trained  by 
their  instructors  to  expect  rapid  responses  through  their  equipment  from 
the  fire  control  system  computer.  Also,  the  students  had  been  trained 
to  depend  on  the  FCO  as  a  system  integrator.  Because  the  FCO  had  to 
wait  until  the  instructor  calculated  the  correct  data  and  inserted  it 
into  the  appropriate  target  data  windows,  he  was  obviously  unable  to  pro¬ 
vide  his  teammates  with  the  information  they  needed.  This  inability  to 


20 


provide  required  information  in  a  timely  manner  completely  obviated 
the  modified  mission  scenario  plan  and  caused  the  performance  of  the 
FCO  and  other  team  members  to  deteriorate  for  these  time-dependent 
tasks. 

A  second  interesting  area  that  was  significantly  affected  by  the 
crew  coordination  training  requirement  was  the  role  of  the  instructor. 
Each  position  had  an  instructor,  and  there  was  an  additional  instructor 
who  served  as  overall  operator  of  the  master  console.  Therefore,  de¬ 
cisions  regarding  pacing,  problem  difficulty,  malfunctions,  emergencies, 
and  system  displays  had  to  be  coordinated  among  several  instructors. 

The  trainer  was  designed  to  be  operated  in  the  crew  coordination  mode 
from  a  set  of  user  created  scenarios  (Cream,  1974) .  Within  these  sce¬ 
narios,  the  instructor  initiated  problems  and  controlled  all  related 
activities.  However,  after  the  trainer  was  put  into  operation,  it  be¬ 
came  clear  that  the  analysis  technique  used  had  not  been  sensitive 
enough  to  detect  a  number  of  subtle  "operational"  techniques  taught  by 
experienced  instructors  in  the  team  context.  The  specific  team  tech¬ 
niques  were  not  part  of  the  normal  syllabus,  nor  were  they  documented 
within  any  of  the  technical  literature.  Yet,  they  seemed  to  be  well 
known  among  the  instructional  staff. 

A  good  example  of  this  situation  can  be  provided  by  continuation 
of  the  discussion  pertaining  to  the  FCO/time  delay  situation.  During 
actual  operation,  the  fire  control  team  members  must  coordinate  their 
target  search  behavior  by  insuring  that  each  of  the  separate  sensor 
operators  is  aware  of  the  areas  being  searched  by  the  other  team  mem¬ 
bers.  This  is  important  for  two  reasons.  First,  target  acquisition 
often  requires  the  coordinated  action  of  more  them  one  sensor.  This  is 
because  different  characteristics  of  the  equipment  itself  make  one  type 
of  target/signal  more  easily  detected  by  one  sensor  than  by  another. 

The  second  reason  that  the  team  members  must  maintain  situational  aware¬ 
ness  is  to  prevent  being  surprised  by  hostile  fire.  Both  of  these  re¬ 
quirements  were  identified  during  the  task  analysis  procedures  prior  to 
trainer  construction.  Some  mission  scenarios  and  target  profiles  were 
set  up  to  exercise  these  skills.  Because  the  instructors  were  all  com¬ 
bat  experienced,  they  had  developed  certain  favorite  techniques  to  use 
during  target  search  and  acquisition.  These  techniques  involved  non¬ 
serial  task/control  actions  that  served  to  accelerate  the  normal  sequence 
of  events.  These  time  compression  techniques  consisted  of  bypassing  task 
steps  and  were  based  upon  instructor  knowledge  of  task  contribution 
versus  the  time  spent  in  performing  the  step.  That  is,  experienced 
instructors  did  not  perform  tasks  with  perceived  low  payoff  for  the  par¬ 
ticular  type  of  target  they  expected  to  find  within  a  specific  situation. 
Depending  upon  the  expected  target,  sensor  priority,  and  aircraft  look 
angle,  steps  within  tasks  were  either  bypassed,  reordered,  or  reduced. 
These  time  compression  techniques  had  not  been  discussed  during  the 
requirement  analysis  phase,  nor  had  they  been  obvious  during  any  of  the 
flights  taken  by  the  training  psychologist  during  task  data  collection. 
Because  of  the  computer  time  delay  problem  at  the  FCO  position,  the 


student  operator  was  unable  to  maintain  the  pace  set  with  use  of  the 
time  compression.  This  inability  in  turn  prevented  the  instructors 
from  demonstrating  some  possibly  critical  skills.  Because  of  the  po¬ 
tential  consequences  of  this  situation,  special  discussions  were  held 
with  students  and  instructor  personnel  to  develop  alternate  training 
techniques.  These  resulted  in  modifications  of  some  mission  scenarios. 
However,  this  solution  was  less  than  totally  satisfactory  for  this 
small  number  of  tasks.  In  spite  of  this  modification,  the  trainer 
did  perform  as  designed.  The  analysis  technique  did  prove  highly  use¬ 
ful  in  identification  of  the  major  issues  involved  in  distinguishing 
crew/tew  coordination  actions  from  those  of  individual  crew  members. 

Although  consideration  of  such  coordination  requirements  would  ap¬ 
pear  to  be  straightforward,  it  is  precisely  the  lack  of  attention  to 
crew  as  opposed  to  individual  training  requirements  that  has  been  cited 
(e.g.,  Faust,  1976)  as  a  major  weakness  of  the  ISD  process.  It  is  ob¬ 
vious  from  experience  with  the  Gunship  trainer  that  failure  to  consider 
team  training  requirements  can  have  serious  implications  for  design  of 
both  the  crew  station  and  instructor  station.  It  is  clear  that  crew 
team  training  cannot  be  conducted  as  an  "add-on"  to  individual  training 
in  a  device  designed  to  limit  fidelity  levels  to  those  required  to  ac¬ 
complish  specific  individual  training  requirements.  It  is  also  clear 
that  an  instructor  station  designed  only  to  permit  control  of  individual 
skill  acquisition  will  not  permit  necessary  coordination  among  individual 
instructors  and  may  not  provide  tin  opportunity  for  instructors  to  exer¬ 
cise  unique  instructional  strategies  that  they  may  adopt  in  a  team  train¬ 
ing  context. 


Gunship  Training  Device  Evaluation 

An  evaluation  of  the  AC-130E  Gunship  trainers  was  conducted  subse¬ 
quent  to  their  design  and  procurement.  In  order  to  evaluate  the  adequacy 
of  the  trainers,  the  performance  of  crew  members  trained  with  the  devices 
was  compared  to  that  of  crew  members  who  had  not  trained  with  the  devices. 
Details  of  the  evaluation  have  been  reported  previously  (Cream  et  al., 

1978).  Basically,  the  purpose  of  the  evaluation  was  to  determine  if  crew 
members  trained  on  the  devices  would;  (1)  achieve  criterion  performance 
with  less  airborne  training  time;  and  (2)  achieve  higher  proficiency  rat¬ 
ings  at  the  completion  of  training  than  controls  who  were  not  trained 
with  the  devices.  Data  concerning  the  acceptability  of  the  devices  for 
training  was  also  gathered  from  the  users.  Because  of  sane  inadvertent 
practice  with  another  trainer  by  members  of  the  FCO  experimental  group, 
that  position  was  excluded  from  further  analysis.  The  evaluation  there¬ 
fore  included  only  the  FLIR,  LLLTV,  and  BC/EWO  positions. 

Results  of  the  evaluation  indicated  that  experimental  groups  reached 
and  maintained  criterion  performance  in  significantly  less  airborne  mis¬ 
sions  than  their  controls.  With  respect  to  proficiency  ratings  at  the 
completion  of  training,  the  BC/EWO  experimental  group  achieved  significantly 
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higher  ratings  than  its  control.  LLLTV  experimental  ratings  were  mar¬ 
ginally  superior  to  controls,  while  FLIR  experimental  group  ratings 
did  not  significantly  differ  from  the  controls.  Overall  user  reaction 
to  training  devices  was  very  favorable,  as  indicated  by  questionnaires 
administered  to  instructors  and  students  who  trained  on  the  device. 

Generally,  the  evaluation  indicated  that  the  trainers  developed 
using  the  techniques  described  above  were  effective  in  providing  train¬ 
ing  at  each  of  the  three  Gunship  positions  that  were  analyzed.  The 
evaluation  also  indicated  that  trainers  were  well  accepted  by  the  in¬ 
structor  and  student  personnel  who  actually  used  the  devices. 


Discussion 


It  is  obvious  from  the  previous  discussion  that  the  technique  has 
been  applied  to  training  device  design  problems  in  predominantly  "machine- 
ascendant"  systems.  "Machine-ascendant"  systems  in  this  context  refer 
to  those  systems  in  which  there  is  a  high  degree  of  man-machine  interac¬ 
tion,  in  addition  to  whatever  interpersonal  interactions  might  occur  in 
the  system  context.  "Man-ascendant"  systems,  on  the  other  hand,  are 
viewed  as  those  in  which  interpersonal  interactions  are  the  primary  sys¬ 
tems  variables. 

The  distinction  drawn  here  between  "machine-ascendant”  and  "man- 
ascendant"  systems  closely  parallels  a  number  of  other  distinctions  that 
have  been  drawn  to  characterize  various  approaches  to  team  training.  One 
of  these  distinctions  is  based  on  different  assumptions  about  the  nature 
of  teams  or  models  of  team  behavior  (Alexander  &  Cooperband,  1965)  .  The 
other  distinction  is  based  upon  points  on  a  continuum  that  can  be  used 
to  characterize  the  task  demands  of  the  situation  in  which  team  behavior 
occurs  (Boguslaw  &  Porter,  1962;  Wagner  et  al.,  1977). 

One  model  of  team  behavior  can  be  characterized  as  a  "Stimulus- 
Response"  (S-R)  model,  while  the  other  is  characterized  as  an  "Organismic" 
model  (Alexander  &  Cooperband,  1965)  .  Team  behavior  in  an  S-R  context 
is  viewed  as  a  series  of  stimulus-response  units.  Team  training,  in  this 
view,  is  an  attempt  to  apply  conditioning  principles  and  other  techniques 
associated  with  individual  S-R  acquisition  to  the  team  (e.g.,  Klaus  & 
Glaser,  1960;  Glazner,  1962).  The  "Organismic"  model,  on  the  other  hand, 
considers  the  team  as  a  synthetic  organism  which  is  composed  of  individual 
team  members  (e.g.,  Kennedy,  1962).  Development  of  the  capability  to 
deal  with  characteristics  of  the  environment  is  viewed  as  a  temporal  pro¬ 
cess.  Team  training  is  therefore  provided  by  establishing  a  synthetic 
environment  in  which  the  development  process  can  take  place. 

These  two  models  can  be  related  to  different  points  on  a  continuum 
of  task  demands  under  which  team  behavior  occurs.  One  extreme  of  the 
continuum  consists  of  established  situations,  the  other  extreme  consists 
of  emergent  situations  (Boguslaw  &  Porter,  1962;  Wagner  et  al.,  1977). 


Established  situations  are  basically  those  in  which  the  environmental 
and  systems  conditions  are  specifiable  and  predictable,  as  are  the 
probable  consequences  of  alternative  actions.  Emergent  situations, 
on  the  other  hand,  are  those  in  which  environmental  conditions  have 
not  been  specified,  systems  states  do  not  conform  to  predictions,  and 
analytic  solutions  to  problems  are  not  available.  It  has  been  sug¬ 
gested  (Alexander  6  Cooper  band,  1965)  that  the  S-R  model  of  team  be¬ 
havior  most  directly  applies  to  teams  operating  in  established  situa¬ 
tions,  while  the  organismic  model  most  readily  applies  to  teams 
operating  in  emergent  situations. 

In  considering  the  applicability  of  the  current  device  design 
technique  to  man-ascendant  and  machine-ascendant  systems,  it  should  be 
noted  that  no  team  works  exclusively  in  an  established  or  in  an  emer¬ 
gent  situation. 

Elements  of  both  situations  appear  in  both  man-ascendant  and 
machine-ascendant  contexts.  One  critical  difference  between  man-  and 
machine-ascendant  systems  may  be  the  degree  of  established  or  emergent 
behavior  that  is  required  in  each. 

The  current  design  technique,  representing  an  outgrowth  and  modi¬ 
fication  of  ISD,  is  certainly  well  equipped  to  deal  with  established 
situations  that  yield  to  a  relatively  straightforward  type  of  S-R  task 
analysis.  Indeed,  much  of  the  technique  is  baaed  on  descriptions  of 
representative  stimulus  configurations  and  required  responses  that  are 
associated  with  the  controls  and  displays  of  the  particular  system  under 
study.  Identification  of  the  specific  team  training  requirements  in  the 
Gunship  trainer  was  based  in  large  measure  upon  the  structured  exchange 
of  information  between  operator  consoles  and  anticipated  channels  of  ver¬ 
bal  communication  between  team  members.  The  Gunship  effort  has  under¬ 
scored  the  importance  of  considering  team  training  requirements  in  design 
of  both  the  trainee  and  instructor  stations  and  has  demonstrated  a  very 
effective  technique  for  doing  so.  Therefore,  to  the  extent  that  either 
man-  or  machine-ascendant  systems  require  behavior  within  established 
situations,  the  current  technique  is  certainly  applicable. 

The  applicability  of  the  current  technique  to  emergent  situations, 
on  the  other  hand,  is  less  clear.  Emergent  situations  typically  involve 
a  high  element  of  decision  making  or  problem  solving,  which  have  tradi¬ 
tionally  not  yielded  well  to  standard  S-R  task  analyses.  It  has  been 
noted  previously  (e.g.,  Klein,  1977;  Cream  et  al.  ,  1978)  that  not  all 
tasks  can  necessarily  be  described  as  fixed  sequences  of  discrete  steps 
and  that  these  tasks  do  not  necessarily  yield  to  S-R  analysis  and  sub¬ 
analysis.  These  considerations  have  become  especially  clear  in  dealing 
with  highly  proficient  performance  and  with  performance  of  highly  com¬ 
plex  tasks.  Experience  with  the  Gunship  trainer  is  consistent  with 
these  observations.  The  one  major  area  that  had  not  been  identified  in 
the  team  analysis  and  that  the  Gunship  trainer  had  not  been  designed  to 
train  was  related  to  time  compression  techniques  that  were  used  by  in¬ 
structors  under  high  task  loading  conditions.  The  time  compression 


techniques  were  highly  dependent  upon  the  specific  context  in  which 
the  overloading  occurred  and  represent  an  excellent  example  of  emer¬ 
gent  behavior. 

It  is  of  some  interest  to  speculate  about  modifications  to  the  cur¬ 
rent  technique  which  might  make  it  more  sensitive  to  identification  of 
team  training  requirements  for  emergent  situations.  One  suggestion  re¬ 
lates  to  a  more  explicit  consideration  of  emergent  situations  during 
the  training  requirement  identification  process.  Although  identifica¬ 
tion  of  specific  behaviors  that  relate  to  particular  stimulus  configura¬ 
tions  would  not  appear  feasible,  it  might  be  possible  to  have  experienced 
proficient  personnel  identify  systems  parameters  that  might  affect  per¬ 
formance  strategies,  etc.  Identification  of  these  parameters  would  take 
place  at  a  more  molar  level  than  traditional  task  analyses,  and  would 
have  the  major  objective  of  directly  specifying  major  systems  variables 
that  should  be  included  in  the  training  simulations.  Such  a  procedure 
would  no  doubt  result  in  higher  fidelity  simulations  them  those  that 
result  from  more  specific  task  analyses.  Such  higher  fidelity  simula¬ 
tions  may,  however,  be  necessary  to  provide  the  opportunity  to  train 
emergent  behaviors.  It  is  obvious  that  development  of  an  analysis  tech¬ 
nique  that  would  extend  the  current  technique  to  emergent  team  training 
situations  needs  to  be  undertaken. 

Another  area  of  consideration  in  structuring  a  team  training  simu¬ 
lation  for  emergent  situations  is  the  instructional  strategies  and  fea¬ 
tures  that  might  be  most  profitably  applied  in  this  context.  Several 
recent  reports  (e.g.,  Wagner  et  al.,  1977;  Kribs  et  al. ,  1977)  have  ad¬ 
dressed  the  issue  of  instructional  strategies  within  team  training. 

This  second  very  important  area  that  will  require  continued  emphasis 
as  a  strategy  for  team  training  device  development  is  refined  to  re¬ 
flect  emergent  situations. 
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4.0  ENGAGEMENT  SIMULATION  TRAINING  SYSTEMS:  SIMULATION 
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TACTICAL  ENVIRONMENTS 
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Dr.  Eddowes  (1978)  has  stated  that  "...  it  is  axiomatic  that  per¬ 
formance  on  a  task  is  facilitated  if  the  acquisition  of  ability  to  per¬ 
form  the  task  is  accomplished  in  a  learning  environment  similar  or 
identical  to  the  environment  in  which  performance  on  the  task  will  be 
ovaluated/measured. "  We  can  make  a  distinction  here  between  "similar" 
and  "identical"  training  environments.  A  learning  environment  identi¬ 
cal  to  the  operational  environment  is  essentially  a  "replication."  In 
a  replication,  there  is,  as  Miller  (1953)  and  Gagne  (1954)  have  pointed 
out,  "engineering  fidelity."  To  achieve  "engineering  fidelity,"  the 
training  system  designer  attempts  to  replicate  the  operational  work  en¬ 
vironment  within  very  tight  tolerance  specification,  "...  whether  or 
not  those  characteristics  provide  critical  cues  or  required  interfaces 
for  the  trainee  learning  a  particular  job"  (Miller,  1975).  Replication 
is  only  possible  when  there  is  sufficient  design  expertise  and  adequate 
technology  available  to  create  a  replica  of  the  operational  environment. 

It  is  possible,  for  example,  to  design  and  build  simulators  for  certain 
electronic  systems  maintenance  jobs  that  are  replicas  of  the  original 
equipment  whose  sole  function  is  training,  such  as  the  Air  Force's  6883 
simulator  that  simulates  the  test  equipment  used  for  trouble  shooting  the 
avionics  of  the  F-lll. 

When  it  is  only  possible  to  approximate  the  characteristics  of  the 
original  equipment  or  operational  environment  due  to  inadequate  design 
expertise  or  technology,  then  the  learning  environment  is  a  "simulation" 
of  the  operational  environment.  The  learning  environment  is  only  similar 
to,  not  identical  to,  the  actual  operational  environment.  For  example, 
the  visual  display  in  most  aircraft  simulators  is  not  a  replica  of  that 
seen  by  the  pilot  through  the  windscreen  of  the  aircraft  ...  it  is  only 
a  simulation  and  therefore  an  approximation  of  what  exists  in  the  real 
world. 

In  simulation,  psychological  fidelity  is  the  critical  variable.  The 
equipment  needs  to  resemble  the  original  or  operational  system  only  to 
the  degree  "...  necessary  for  the  trainee  to  transfer  to  the  job  environ¬ 
ment  .  .  .  those  responses  he  learns  to  perform  in  the  training  environ¬ 
ment"  (Miller,  1975).  The  basic  notion  here  is  that  given  the  trainee 
learns  to  emit  a  set  of  responses  (Rj._j)  to  a  set  of  stimuli  (Si_j)  in  a 
training  environment,  there  is  a  set  of  stimuli  in  the  operational  en¬ 
vironment  (Sk-i)  that  psychologically  "maps"  on  to  the  set  (Si— j )  such 
that  (Ri_j)  is  also  elicited  by  (S^.^) .  (Sk_i>  psychologically  "maps" 
on  to  the  set  (S^_j)  by  occurring  well  within  the  generalization  gradient 
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Probability  that  (R: ;)  will  ba  did  tad  by  •  stimulus  of  type  "S" 


of  (Sj.j) 
detect abl 


so  that  there  1b  only  a  minimal,  insignif leant  discrepancy 
la  in  the  probability  that  (Ri_j)  will  be  emitted  in  the  presence 
of  (S^. )  ,  as  shown  in  Figure  4-1.  The  discrepancy  between  P  actual  and 
P  simulation  can  be  called  the  "Transfer  of  Training  Discrepancy,"  or 
TOT^,  for  the  response  repertoire  being  trained. 


^simulation  minus  ^actual 


Sp  (The  set  of  all  possible  stimuli.) 


Figure  4-1. 


Generalisation  gradient  for 


We  can  say  that  psychological  fidelity  increases  when  TOTA  decreases. 
TOTa  is  an  indication  of  the  extent  to  which  the  set  of  actual  stimuli  ap¬ 
proximates  the  set  of  stimuli  that  occur  in  the  simulated  environment, 
which  are  known  to  elicit  the  target  response  set  with  some  probability 
exhibited  during  training.  The  extent  of  this  convergence  depends  on  a 
number  of  factors,  not  the  least  of  which  is  individual  variation  in 
generalization  gradients.  But  it  has  been  possible  to  build  training 
environments  where  the  average  TOTA  is  sufficiently  small  and  the  indi¬ 
vidual  variability  reasonably  controlled  to  yield  a  highly  cost-  and 
training-effective  approach  for  a  training  population. 
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It  is  important  to  note  that  the  convergence  of  (S^-i)  to  (Sj.)  , 
and  consequently  the  size  of  TOT^,  is  not  necessarily  a  function  of 
engineering  fidelity.  Eddowes  (1978)  ,  for  example,  lias  found  that  vari¬ 
ous  increases  in  engineering  fidelity  in  the  Advanced  Simulator  for 
Pilot  Training  (ASPT)  have  had  little  or  no  impact  on  certain  performance 
variables  studied.  The  same  conclusion  was  reached  by  Martin  and  Waag 
(1978)  in  their  work  with  the  ASTT.  Furthermore,  Eddowes  found  that  in 
one  case,  the  system  with  a  lower  level  of  engineering  fidelity  yielded 
better  in  simulator  performance  of  various  flying  maneuvers  by  inexperi¬ 
enced  pilots. 

The  point  here  is  that  it  does  not  appear  unreasonable  to  consider 
psychological  and  engineering  fidelity  as  different  dimensions  in  any 
replication  or  simulation.  Depending  on  the  simulation,  psychological 
and  engineering  fidelity  are  related  by  a  rather  complex  function.  Fol¬ 
lowing  from  this  assumption,  increments  in  engineering  fidelity  do  not 
necessarily  decrease  TOT^.  Generally  speaking,  then,  there  may  bo  sepa¬ 
rate  continua  of  psychological  fidolity  and  physical  fidelity  in  any  given 
simulation  whose  precise  functional  relationship  may  vary  from  simple  to 
complex,  dependent  upon  various  characteristics  of  the  operational  envi¬ 
ronment  being  simulated.  Precisely  what  are  these  characteristics  and 
the  manner  in  which  they  affect  this  relationship  is  a  matter  for  specu¬ 
lation  and  perhaps  research.  With  regard  to  psychological  or  behavioral 
fidelity,  however,  there  is  a  relatively  simple  first  order  relationship 
between  psychological  fidelity  (the  relative  psychological  contiguity  of 
actual  and  simulated  stimulus  environments)  and  TOT^. 

This  distinction  between  psychological  and  engineering  fidelity  is 
clearly  reflected  in  the  range  of  simulation  training  systems  developed 
by  the  Department  of  Defense.  "Representing  the  upper  bounds  of  current 
simulation  technology  is  the  sophisticated  device  which  reproduces,  to 
the  greatest  extent  possible,  the  complete  capability  and  environment  of 
a  weapon  system  for  a  specific  mission"  (Allen,  1977).  Miller  (1975) 
calls  this  the  "full-engineering"  approach  in  which  little  or  no  atten¬ 
tion  ".  .  .  is  paid  to  the  psychological  factors  of  training  equipment 
design.  .  that  is,  little  or  no  attention  to  psychological  fidelity 
and  the  factors  that  can  affect  the  psychological  "proximity"  of  the 
target  set  of  stimuli  from  the  operational  environment,  (S^_^)  ,  to  those 
of  the  simulated  stimuli,  (S^_j). 

It  is  not  difficult  to  understand  how  the  full-engineering  approach 
could  overwhelm  the  trainee,  leading  to  results  such  as  those  described 
by  Eddowes  (1978)  and  Martin  and  Waag  (1978)  for  the  ASFT.  Any  number 
of  factors,  ranging  from  premature  loading  of  the  attentional  capacity  of 
the  trainee  to  improperly  sequenced  learning  "challenges,"  could  render 
the  "full-engineered"  simulator  less  effective  in  eliciting  desired  levels 
of  performance.  The  solution  to  this  sort  of  training  problem  often  lies 
in  some  abstraction  of  the  job  specifically  tailored  to  optimizing  the 
acquisition  of  a  limited  response  repertoire. 
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"At  the  lower  end  of  the  equipment  capability  spectrum  are  those 
devices  which  simulate  the  actual  equipment  only  as  far  as  external  con¬ 
figuration  .  .  (Allen,  1077).  Those  "partial  engineering"  devices 
are  designed  for  part-task  training  and  may  or  may  not  include  considera¬ 
tion  of  psychological  factors  critical  for  achieving  psychological  fidel¬ 
ity.  Devices  such  as  maintenance  simulators  and  part-task  flight  simu¬ 
lators  would  represent  this  category  as  opposed  to  full  mission  simulators 
such  an  tho  ASPT. 

Beyond  the  bounds  of  Allen's  equipment  capability  spectrum  is  DOD's 
rocent  interest  in  the  development  of  combat  engagement  simulation  tech¬ 
niques.  With  these  systems,  whose  objective  is  to  simulate  combat,  engi¬ 
neering  fidelity  is  of  less  concern  than  psychological  or  behavioral 
fidelity.  The  reason  for  this  fact  lies  in  the  difference  between  the 
relative  importance  of  tho  man-machine  system  that  comprises  the  cockpit 
and  fire  control  system  of  a  jet  fighter  and  that  which  exists  in  a  com¬ 
bat  unit.  Emphasis  on  "full-engineering”  solutions  has  been  tho  case  for 
primarily  "machine-ascendant"  systems  such  a3  flight  simulators  and  elec¬ 
tronic  maintenance  trainers  in  which  tho  performance  requirements  are  de¬ 
rived  from  the  operator's  interaction  with  the  machine,  via  an  analytic 
framework  such  as  Task  Analysis.  Since  the  training  objective  is  usually 
to  get  the  operator  to  omit  appropriate  responses  to  stimuli  emanating 
from  the  man-machine  interface,  tho  most  effective  reduction  in  TOT^  comes 
from  attempting  to  approximate  the  physical  characteristics  of  the  system 
(that  is,  the  machine's  characteristics)  as  clearly  as  possible.  But  in 
a  man-ascendant  system,  in  which  the  variability  of  system  performance  is 
predicted  in  a  significantly  greater  dogroe  by  performance  in  man-man 
interfaces,  the  emphasis  has  been  naturally  on  psychological  variables 
and  consequently  on  achieving  psychological  fidelity  in  a  simulated  work 
environment. 

In  man-ascendant  systems,  critical  cuos  and  stimuli  are  to  be  found 
in  the  collective  dynamics  of  the  team  or  unit,  particularly  for  those 
skills  that  involve  team  or  unit  effort  and  coordination.  As  Dr.  Eggemeier 
(1978)  lias  pointed  out,  tasks  that  appear  to  be  of  little  importance  when 
performed  individually  can  become  of  critical  importance  when  performed 
in  tho  context  of  a  team  striving  to  achieve  collective  goals  or  strate¬ 
gies.  It  is  tho  "context"  of  toam  functioning  that  contains  tho  stimuli 
that  have  to  bo  replicated  or  simulated  in  the  training  environment. 

It  is  not  an  easy  matter  to  analyse  tho  "context"  of  team  performance 
to  isolate  the  critical  cuos  and  stimuli  required  for  a  high  psychological 
fidelity  simulation  training  system,  particularly  in  man-ascendant  sys¬ 
tems.  Man-ascendant  systems  pose  unique  analytic  problems  for  a  number 
of  reasons.  In  man-ascondant  systems,  there  is  the  possibility  of  fol¬ 
lowing  a  systematic  method  for  analysing  the  tasks  of  the  operators, 
choosing  those  that  are  critical  for  training,  and  then  designing  a  simu¬ 
lator  to  fit  tho  bill.  Methods  such  as  Goldstein's  (1974),  or  the  Air 
Force's  (USAF  Manual  50-2,  1970:  USAF  Pamphlet  50-58,  1974),  allow  the 
training  system  designer  to  isolate  which  displays,  consolos,  knobs, 
situations  should  bo  replicated  or  simulated  for  training  the  operator. 
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The  designer  can  then  sit  down  with  the  engineer  and  determine  the  ex¬ 
tent  to  which  the  portion  of  the  operational  environment  chosen  for 
training  can  be  replicated  or  simulated.  The  main  stumbling  block 
left  at  this  point  is  the  technology  available  for  building  the  repli¬ 
cation.  Is  there  a  CRT  display  big  enough,  or  a  platform  motion  system 
that  can  generate  X  g's  within  Y  seconds  in  ”  directions,  for  example. 
Accompanying  the  task  analytic  approach  is  usually  an  explicit  descrip¬ 
tion  of  what  is  to  be  trained,  the  condition  under  which  mastery  should 
bo  demonstrable,  and  the  minimum  level  of  acceptable  performance,  as 
Smodc  (1971)  has  described. 

The  ISD  approach  is  particularly  well  suited  to  individual  train¬ 
ing  situations  in  machine-ascendant  work  environments.  It  may  also  be 
applicable  to  team  or  crew  training  situations,  as  TAEG  Report  1H  (1975) 
argues.  However,  it  would  only  seem  to  be  applicable  in  certain  cases 
in  which  system  performance  is  more  a  function  of  individual-machine  in¬ 
teraction  as  opposed  to  man-man  or  man-crew  interaction.  "...  Tasks 
team  members  are  required  to  perform  (most  notably,  the  team-interaction 
tasks)  in  an  operational  context  (can)  be  identified  and  carefully  ana¬ 
lyzed,"  the  TAEO.  report  states.  However,  tills  approach  may  be  somewhat 
simplistic,  and  to  a  certain  extent  erroneous,  in  that  some  of  the  most 
critical  aspects  or  characteristics  of  team  performance  may  only  emerge 
durinq  team  interactions,  the  dynamics  of  which  are  not  readily  reduci¬ 
ble  to  the  framework  of  a  task  analysis.  Consequently,  training  systems 
designed  to  teach  crew  or  team  behavior  may  lack  those  situations  (the 
complexes  of  stimuli  and  cues)  required  for  the  elicitation  of  critical 
team  skills.  Alexander  and  Cooperband  (1965)  have  argued  that,  often  in 
team  training,  the  tasks  being  trained  are  such  that  exhaustive  formal 
rules  cannot  be  stated.  Teams  often  develop  procedures  and  behaviors 
in  the  process  of  accomplishing  a  task;  consequently,  it  is  often  only 
feasible  to  provide  teams  with  situations  where  they  must  innovatively 
and  effectively  develop  procedures. 

More  recently,  a  number  of  authors  have  argued  that  for  team 
training,  the  ISD-task  analytic  methodology  does  not  work.  Faust  (1976) 
concludes  that  "...  unfortunately,  present  ISO  methodology  as  generally 
documented  is  often  weak  in  techniques  which  are  suitable  to  the  identi¬ 
fication,  design,  or  validation  of  components  which  require  team  train¬ 
ing."  Faust  feels,  however,  that  innovative  ISD  techniques  can  "...  of 
for  considerable  promise  ...  if  ISD  is  seen  as  a  dynamic  and  responsive 
process."  This  opinion  is  somewhat  more  optimistic  than  that  offered  by 
Klein  (1976)  who  feels  that  the  ISD  and  Systems  Approach  to  Training 
(SAT),  as  promulgated  by  the  Air  Force,  have  some  basic  weaknesses  that 
make  these  approaches  unsuitable  for  simulation  design,  particularly 
where  complex  individual  and  crew  tasks  are  involved.  Cream  (1975) 
felt  that  the  general  family  of  ISD  methodologies,  typified  by  AF  Manual 
50-2  and  AF  Pamphlet  50-58,  "...  don't  adequately  address  the  complex 
issue  of  crew  coordination."  Cream  also  concludes  that  when  system 
performance  depends  on  team  coordination,  task  analysis  "...  will  not 
be  adequate,  and  may  possibly  be  misleading."  Cream,  Eggemeier,  and 
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Klein  (1978)  have  offered  an  alternative  methodology;  the  Criticality, 
Frequency  of  performance,  and  Difficulty  of  performance  (CFD)  frame¬ 
work  provides  an  interesting  alternative  to  conventional  1SD  methodol¬ 
ogy,  and  one  that  is  more  oriented  toward  an  effective  assessment  of 
the  very  complex  dynamics  that  characterise  crew  and  team  activities. 


But  there  are  simulation  design  problems  for  crew  or  team  activi¬ 
ties  that  may  not  be  amenable  to  this  sort  of  analysis  at  any  level. 

Dr.  Eggemeier  touched  on  this  problem  in  his  closing  remarks  in  stating 
that  the  analysis  of  tasks  and  activities  in  team  or  unit-based  opera¬ 
tional  environments  should  be  terminated  at  a  relatively  global  level. 
Cream,  Eggemeier,  and  Klein  (1975)  mentioned  that  such  behavior  measured 
as  a  totality  or  at  a  global  level  may  be  too  gross  for  training  analy¬ 
sis  or  design  purposes  .  .  .  but  the  same  behavior  measured  as  discrete 
steps  ignores  the  situational  or  "Gestalt"  qualities  of  the  activity. 
Thus  the  dilemma. 


The  point  being  made  here  is  that  recently  made  by  Magner,  Hibbits, 
Rosenblatt,  and  Schulz  (1977) ,  in  their  review  of  team  training  and  evalu¬ 
ation  strategies:  there  are  two  approaches  to  conceptualizing  team  be¬ 
havior,  especially  for  designing  training  strategies.  First,  team  activi¬ 
ties  can  be  thought  of  as  multi-individual  activities  in  which  individuals 
perform  established  procedures  which  are  amenable  to  S-R  analyses.  Those 
established  procedures  are  ex  Vicit,  probably  written,  and  can  be  taught 
on  an  individual  basis.  For  cne  crew  to  function  effectively,  each  indi¬ 
vidual  need  only  perform  his  or  her  tasks  effectively.  In  this  sense, 
the  crew  output  represents  the  sum  of  individual  activities.  The  second 
approach  is  that  of  conceptualizing  the  team  as  an  organismic  entity  in 
which  the  activities  of  the  individuals  are  mutually  interdependent,  and, 
which  are  defined  and  instigated  not  only  in  terms  of  one  another's  ac¬ 
tivities  but  in  terms  of  the  activity  of  the  team,  per  se.  Individuals 
working  in  an  organismic  team  or  crew  operational  environment  perform 
tasks  which  are  difficult  to  predict  and  analyze.  These  tasks  or  activi¬ 
ties  can  be  considered  as  emergent  phenomena  in  that  they  arise  from  com¬ 
plex  situations  which  are  characterized  by  equally  complex  arrays  of 
stimuli  whose  occurrence  is  probabilistic  rather  than  deterministic,  as 
far  as  the  application  of  analytic  techniques  might  be  concerned.  Fur¬ 
thermore,  the  response  repertoires  which  occur  in  these  situations  can 
also  be  considered  to  be  emergent  in  that  the  probability  that  the  exact 
response  will  occur  twice  is  very  low  due  to  the  very  low  probability 
that  the  exact  stimulus  situation  will  develop  repeatedly. 


In  such  a  situation,  the  trainee  does  not  usually  need  to  learn 
procedural  aspects  of  the  job  and  the  operational  environment,  but  rather 
the  critical  training  requirement  is  usually  the  third  phase  of  team 
training  described  by  Kanarick,  Alder.,  and  Daniels  (1971)  in  which  teams 
are  taught  to  apply  their  ".  .  .  procedural  and  interactive  skills  to 
.  .  .  situations  requiring  innovative  and  creative  behavior."  The  team 
training  requirement  in  this  case  is  less  that  trainees  need  to  learn 
specific  responses  and  more  that  the  individuals  in  the  team  learn  to 
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quickly  and  innovatively  access  diverse  response  repertoires  in  re¬ 
sponse  to  and  in  support  of  team  dynamics  and  goals.  Kanarick  et  al. 
(1971)  speak  of  "team  awareness":  this  is  the  sort  of  milieu  or  at¬ 
mosphere  in  which  is  often  the  training  objective  for  teams  working  in 
these  sorts  of  unusual,  complex  environments. 

For  operational  environments  in  which  system  performance  is  pre¬ 
dicted  primarily  by  the  capabilities  of  individuals  for  effectively 
assessing  diverse  though  appropriate  response  repertoires  to  emergent, 
unpredictable  stimulus  environments,  it  would  seem  that  any  sort  of 
analysis,  global  or  otherwise,  is  really  not  practical  or  cost-effective. 
Dr.  Eggemeier's  point  that  ".  .  .a  global  procedure  which  includes  the 
C/F/D  analysis  could  be  an  initial  step  in  a  solution  of  the  fidelity 
specification  problem  in  'man-ascendant'  systems"  is  well  taken  here. 

But  analysis  of  the  operational  environment  is  by  definition  a  very  dif¬ 
ficult  problem  when  the  actual  occurrence  of  various  stimulus  environments 
is  relatively  unpredictable  and  for  which  there  are  no  unique  "correct" 
or  "criterion"  response  repertoires.  Perhaps  some  sort  of  global  analy¬ 
sis  which  merely  identifies  the  operational  environment  as  one  composed 
of  emergent  situations  is  the  analytic  tool  needed! 

The  fidelity  problem  here  is  particularly  difficult.  How  does  the 
training  system  designer  provide  a  set  of  stimuli  that  are  designed  to 
clearly  resemble  those  stimuli  in  the  operational  environment  when  it  is 
not  possible  to  isolate  stimuli  which  consistently  elicit  target  response 
repertoires?  And  if  the  correct  response  is  situation  dependent,  and 
situations  are  emergent,  idiosyncratic  phenomena,  how  does  the  designer 
specify  behavioral  objectives  from  which  stimulus  environments  can  theo¬ 
retically  be  designed?  And  furthermore,  all  these  machinations  occur 
in  the  realm  of  psychological  fidelity  in  which  one  of  the  principal 
training  concerns  is  team  awareness  and  coordination. 

A  very  cogent  example  of  this  problem  area  and  one  in  which  the  Army 
Research  Institute  has  worked  for  several  years  is  the  design  of  training 
simulations  for  combat.  Root,  Knerr,  Severino,  and  Word  (1978)  have  dis¬ 
cussed  combat  engagements  as  examples  of  complex,  man-ascendant  systems 
in  which  the  manner  and  speed  with  which  a  task  is  accomplished  is  situ- 
ationally  determined.  The  situation  is  driven,  in  part,  by  an  intelli¬ 
gent  adversary  as  well  as  by  less  critical  inputs  such  as  terrain,  rela¬ 
tive  force  strength,  and  weather.  Root  et  al.  make  the  point  that 
".  .  .  in  the  mem-ascendant  system  represented  by  a  combat  unit,  there 
can  be  a  number  of  possible  routes  to  the  system's  goal";  in  this  sort 
of  system,  the  man  is  more  a  decisionmaker  than  an  operator  and  often 
any  of  a  number  of  decisions  are  the  correct  solution  to  the  situation. 
Furthermore,  each  decision  to  act  and  act  in  a  certain  way  represents 
an  individual's  response  to  not  only  a  range  of  situational  cues  provided 
by  the  enemy  but  also  a  complex  evaluation  of  behavioral  options  in  which 
the  individual  must  be  aware  of  his  role  in  the  unit,  the  unit's  overall 
mission  and  goal  in  each  situation,  and  the  unit's  activities  as  a  col¬ 
lective  entity. 
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ARI  has  designed  engagement  simulation  training  systems  for  a 
variety  of  different  types  of  combat  situations  involving  infantry, 
armor,  armored  cavalry,  air  defense,  and  aviation.  As  Root  et  al.  have 
discussed,  it  has  not  been  possible  to  task  analyze  combat:  "When  con¬ 
sidering  the  necessary  interactions  among  unit  elements  .  .  .  is  it 
really  practical  to  assume  that  'all  critical  processes  involved  in 
successful  mission  accomplishment'  can  be  identified  by  the  task  ana¬ 
lytic  process?”  Consequently,  the  design  problem  has  been  unique,  dif¬ 
ficult,  and  has  required  a  good  deal  of  innovative  effort  by  the  staff 
of  the  Engagement  Simulation  Project. 

In  the  simulation  of  combat,  an  initial  assumption  is  made  that 
it  is  not  possible  to  specify  a  set  of  responses  (R^_ j )  for  any  given 
set  of  stimuli  that  may  occur  in  the  operational  environment.  That  is, 
there  is  no  specific  response  repertoire  that  is  correct,  right,  or 
necessarily  desirable  for  a  given  set  of  tactical  stimuli.  Instead, 
the  training  objective  is  to  have  units  develop  the  capability  for  in¬ 
novative  and  effective  response  repertoire  selection  to  tactical  situa¬ 
tions.  Root  and  Erwin  (1977),  in  an  attempt  to  operationally  define 
high  proficiency  combat  performance,  have  stated  that  one  of  the  prin¬ 
ciple  characteristics  is  effective,  adaptive  response  to  enemy  dynamics, 
in  addition  to  maximizing  inter-  and  intra-unit  coordinations,  and 
maximizing  friendly  fire  power  while  minimizing  the  effect  (or  potential 
for  effect)  of  enemy  fire  power.  Consequently,  one  of  the  design  re¬ 
quirements  for  engagement  simulation  training  environments  is  to  allow 
sufficient  freedom  in  the  simulation  for  the  trainee  and  team  to  choose 
among  response  repertoires,  and  to  provide  constantly  varying  stimulus 
situations  so  that  trainees  will  be  able  to  develop  and  practice  selec¬ 
tion  of  response  repertoires  to  complex,  idiosyncratic,  and  situational 
tactical  stimulus  environments. 

In  engagement  simulation,  this  objective  is  realized  by  having 
two-sided,  free-play  combat  engagements  in  which  the  "enemy"  is  training 
as  well.  In  this  way,  situations  unfold  that  are  thoroughly  the  result 
of  the  collective  dynamics  of  the  two  teams  in  the  simulation.  These 
situations  are,  as  Boguslaw  (1961)  and  Boguslaw  and  Porter  (1962)  de¬ 
scribe,  purely  emergent  situations:  (1)  all  action-relevant  environmental 
conditions  cannot  be  predicted  ahead  of  time;  (2)  the  state  or  output 
of  the  system  at  any  given  moment  is  not  accurately  predictable  with 
high  probability;  and  (3)  the  situations  are  not  readily  amenable  to 
analysis  vis  A  vis  the  current  state  of  analytic  technology. 

Although  the  development  and  prediction  of  these  emergent  situa¬ 
tions  is  presently  beyond  the  capabilities  of  the  network  of  control  per¬ 
sonnel  used  to  run  engagement  simulation  exercises,  research  is  being 
done  on  developing  analytic  frameworks  for  understanding  the  dynamics 
of  such  phenomena.  Hamill  (1978)  and  Epstein  (1978)  have  described  an 
Automated  Tactical  Operations  Measurement  System  (ATOMS)  that  is  de¬ 
signed  to  build  a  data  base  from  engagement  simulations  exercises  that 
will  yield  correlations  between  "process"  activities  of  teams  (such  as 
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movement  and  communication)  and  "product"  measures  (such  as  casualty- 
exchange  ratios  and  achievement  of  objectives)  .  High  resolution,  compre¬ 
hensive  data  collection  systems,  such  as  ATOMS,  whose  analytic  method  is 
based  on  identifying  internal  consistencies  within  exercises  that  seem 
to  be  repeated  between  exercises,  may  prove  to  be  the  first  step  in  docu¬ 
menting  and  understanding  which  complex  behaviors  in  emergent  situations 
are  to  be  considered  "good"  or  "desirable."  This  crucial  first  step  is 
necessary  to  identify  performance  objectives  for  simulation  of  collec¬ 
tive  man-ascendant  systems. 

The  present  inability  to  analyze  and  document  these  behaviors  makes 
it  particularly  difficult  to  develop  or  design  an  engagement  simulation 
training  system.  To  date,  the  design  strategy  of  the  Army  Research  In¬ 
stitute's  Tactical  Engagement  Simulation  Project  has  been  to  insure  that 
certain  basic  design  guidelines  are  met  that  have  proven  successful  in 
initially  designing  "face-valid"  simulation  systems  that  have  later  been 
formally  validated  by  field-experimentation.  First,  as  mentioned  ear¬ 
lier,  an  effort  is  made  to  insure  that  there  are  sufficient  opportunities 
for  teams  to  have  to  innovatively  and  effectively  access  various  response 
repertoires  as  emergent  stimulus  environments  occur.  To  accomplish  this, 
care  is  taken  that  the  exercise  scenario  or  setting  will  support  a  com¬ 
pletely  free-play,  two-sided,  unprogramed  interaction.  Second,  the  ex¬ 
ercise  architects  seek  to  achieve  a  motivational  level  that  will  drive 
a  free-play  atmosphere  by  building  in  the  same  behavioral  contingencies 
that  the  trainee  has  been  taught  exist  in  combat.  For  example,  near 
real-time  casualty  assessment  appears  to  be  one  of  the  most  critical  of 
these  contingencies.  If  the  individual  or  unit  knows  that  they  can  be¬ 
come  a  casualty  and  can  inflict  casualties,  then  it  is  possible  for  them 
to  perceive  the  training  as  relevant  to  their  survival  in  combat.  The 
development  of  credible  contingencies  in  the  simulation  motivates  the 
troops  to  the  point  where  the  collective  dynamics  that  generate  emergent 
situations  are  adequately  driven  and  fostered. 

Third,  for  each  type  of  exercise,  weapon  signatures  are  simulated 
as  well  as  the  terminal  effects  of  various  weapons  by  using  pyrotechnics 
and  various  electromechanical  devices.  An  attempt  is  made  to  provide 
the  sights  and  sounds  of  battle,  partly  to  provide  the  same  contingen¬ 
cies  as  in  combat  (e.g.,  if  the  firing  of  your  weapon  system  discloses 
your  position,  you  can  then  be  engaged)  and  partly  to  heighten  psycho¬ 
logical  fidelity  by  including  the  violent  distr actors,  unexpected  con¬ 
cussions,  and  associated  events  chat  provide  the  stressful,  disorganiz¬ 
ing,  attentional  load  that  can  occur  in  combat. 

As  is  clearly  evident,  it  is  not  possible  to  task  analyze  combat; 
it  is  not  possible  to  specify  performance  objectives  because  there  are 
no  one-to-one  correspondences  between  stimuli  and  desired  responses; 
and  it  is  not  possible  to  program  activities  since  the  learning  situa¬ 
tions  desired  only  arise  from  the  dynamics  of  free-play  interactions. 

The  design  goal,  then,  is  to  give  trainees  the  opportunity  to  function 
in  a  combat  environment,  and  to  give  the  trainee  the  opportunity  to 


chango  that  environment  as  a  function  of  their  actions,  contributing 
to  collective  dynamics. 

Designers  of  engagement  simulation  training  systems  have  evaluated 
the  fidelity  and  validity  of  the  exercises.  And  it  is  in  this  activity 
that  there  is  a  good  deal  of  relevance  to  Dr.  Matheny's  (1978)  perfor¬ 
mance  equivalence  model.  On  two  occasions  in  1977  and  1978,  the  Army 
Research  Institute  has  formally  validated  engagement  simulation  training 
systems.  These  activities  have  been  described  in  two  reports  by  Banks, 
Hardy,  Scott,  Knerr,  and  Word  (1977),  and  Meliza,  Scott,  and  Epstein 
(1978) .  The  purpose  of  these  validations  was  to  demonstrate  that  engage¬ 
ment  simulation  provides  a  training  environment  for  exercising  and  learn¬ 
ing  collective  tactical  skills,  and  that  engagement  simulation  training 
supports  more  efficient  acquisition  of  these  skills  than  conventional 
tactical  training  activities. 

Dr.  Matheny  (1978)  describes  "...  the  performance  (that)  we  are 
interested  in  .  .  .  being  the  metric  for  measuring  fidelity."  In  validat¬ 
ing  engagement  simulation  training  systems,  a  very  similar  rationale  is 
used.  For  field-validating  these  training  systems,  data  are  collected 
that  document  behavior  which  are  consistently  exhibited  both  within  and 
among  exercises.  These  behaviors  can  then  be  judged  as  similar  to  or  dis¬ 
parate  from  those  that  the  training  system  proponents  would  hope  would  be 
exhibited  in  combat.  For  example,  proponents  for  engagement  simulation 
tactical  training  for  armor  and  infantry  teams  would  expect  that  field 
exercises  would  provide  the  proper  setting  for  eliciting  effective  com¬ 
munication  and  coordination  between  armor  and  infantry  leaders.  Further¬ 
more,  tank  platoons  would  be  expected  to  utilize  cover  and  concealment 
techniques  that  follow  the  most  recently  published  Army  doctrine.  Data 
collected  during  repeated  exercises  on  communication  rates,  communicating 
dyads,  and  communication-initiated  action  can  be  used  to  examine  whether 
or  not  "combat-like"  behaviors  are  elicited  by  the  simulated  tactical 
environment.  Similarly,  data  collected  on  the  locations  of  light  and 
heavy  tank  sections  relative  to  available  terrain  can  be  used  to  judge 
whether  tank  platoons  use  cover  and  concealment  in  response  to  the  simu¬ 
lated  tactical  situation. 

Furthermore,  it  is  possible  to  evaluate  the  criticality  of  various 
behaviors  to  general  combat  proficiency.  The  frequency  of  occurrence  of 
behaviors  and  the  level  of  proficiency  at  which  they  appear  to  be  ex¬ 
hibited  can  be  correlated  with  total  system  output,  such  as  winning  the 
problem,  or  operating  at  a  particular  casualty  exchange  ratio.  In  this 
way,  not  only  can  the  system  be  validated,  so  to  speak,  but  it  can  be 
evaluated  by  determining  if  it  provides  a  stimulus  environment  that  not 
only  elicits  correct  behaviors  but  also  provides  the  opportunity  for  de¬ 
velopment  of  proficiency. 


Data  have  also  been  collected  to  determine  if  valid  collective 
tactical  behaviors  consistently  observed  in  engagement  simulation  ex¬ 
ercises  are  exhibited  at  a  higher  level  of  proficiency  in  a  shorter 
period  of  time  than  if  conventional  training  had  been  used.  The  gen¬ 
eral  finding  has  been  that  engagement  simulation  trains  proficient  col¬ 
lective  tactical  behaviors  faster  than  conventional  training  techniques. 
The  methodology  that  has  been  followed  is  to  obtain  baseline  data  by 
having  two  units  engage  in  series  of  exercises,  then  having  one  unit 
train  with  conventional  procedures  while  the  other  unit  continues  to 
conduct  engagement  simulation  exercises,  and  then  bringing  the  two 
units  together  for  a  final  series  of  exercises  in  which  performance  can 
be  compared.  This  procedure  allows  for  the  collection  of  definitive 
data  on  the  training  efficiency  of  the  engagement  simulation  training 
procedures  used:  Does  the  simulation-trained  team  do  better  than  the 
conventionally  trained  team  in  what  is  considered  a  valid  test  bed  for 
collective  tactical  skills  (i.e.,  the  simulated  combat  environment)? 

In  summary,  what  are  the  basic  characteristics  of  engagement  simu¬ 
lation  training  systems  that  pose  unusual  problems  for  the  training 
simulation  designer? 

1.  By  virtue  of  the  operational  environment  being  simulated, 
the  design  emphasis  of  the  simulation  has  to  be  psychologi¬ 
cal,  as  opposed  to  physical,  fidelity. 

2.  A  combat  unit  is  a  man-ascendant  system,  in  which  both  in¬ 
dividual  and  collective  behavior  occur  in  emergent  situations. 
These  emergent  situations  are  not  predictable  and  are  diffi¬ 
cult  to  analyze.  Consequently,  it  is  not  possible  to  design 

a  simulation  that  yields  the  same  stimulus  environment  and 
which  elicits  the  same  response  repertoires  repeatedly. 

3.  Engagement  simulation  is  designed  to  allow  for  the  develop¬ 
ment  and  occurrence  of  emergent  stimulus  environments  for 
which  the  individual  or  unit  must  choose  an  appropriate  and 
effective  response  repertoire. 

4.  The  psychological  fidelity  of  engagement  simulation  systems 
is  measured  by  the  extent  to  which  the  simulated  environment 
elicits  effective  combat  behaviors.  Because  it  is  not  possi¬ 
ble  to  task  analyze  combat  due  to  the  emergent  and  incredibly 
complex  nature  of  unfolding  events,  it  is  necessary  to  show 
that  elicited  behaviors  are  related  to  total  system  perfor¬ 
mance,  such  as  winning  or  losing  a  battle,  or  incurring  "X" 
casualties  while  inflicting  "Y"  casualties.  This  demonstra¬ 
tion,  that  elicited  behaviors  are  related  to  total  system 
performance,  is  a  research  and  analysis  problem  currently 
being  tackled  by  ARI  staff. 
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Combat  has  proven  to  be  a  difficult  challenge  for  the  sinulation- 
oriented  training  system  designer.  But  as  John  Allen  (1977),  Deputy 
Director  for  Research  and  Advanced  Technology  in  DDRE,  has  stated, 

”...  the  payoff  for  the  military  may  be  just  as  great  for  combat 
engagement  simulation  .  .  .  as  it  is  for  flight  simulation."  And  the 
payoff  for  flight  simulation  has  been  too  great  not  to  apply  this  ap¬ 
proach  to  man-ascendant  collective  training  problems,  such  as  combat. 

Awareness  of  the  unique  requirements  facing  the  training  system 
designer  working  with  collective,  man-ascendant  environments  is  critical 
in  building  a  valid  simulation.  Too  often  the  task-analysis  based  ISD 
approach  has  been  applied  to  these  sorts  of  operational  environments  to 
the  detriment  of  the  validity  and  fidelity  of  the  training  simulation. 

The  very  important  collective  behaviors  and  capabilities  that  are  elic¬ 
ited  by  and  practicable  in  emergent  situations  end  up  not  being  trained. 
And  for  many  of  these  systems,  these  sorts  of  behaviors  account  for  the 
greatest  portion  of  the  system's  variability.  Whether  it  is  a  question 
of  using  simulation  to  train  combat  units,  management  teams,  or  elemen¬ 
tary  social  skills  in  therapeutic  or  educational  environments,  the  same 
unique  emergent  characteristics  of  dynamic  collective  systems  are  present. 
And  to  ignore  these  characteristics  by  using  analytic  techniques  that 
cannot  describe  or  provide  design  guidelines  for  emergent  learning  situ¬ 
ations  is  a  serious  failing  on  the  part  of  the  training  system  architect. 
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5.0  FLIGHT  SIMULATOR  FIDELITY  AND  FLYING 
TRAINING  EFFECTIVENESS 


Edward  E.  Eddowes 
Flight  Training  Division 
Air  Force  Human  Resources  Laboratory 


It  is  axiomatic  that  performance  on  a  task  is  facilitated  if  the 
acquisition  of  ability  to  perform  the  task  is  accomplished  in  a  learn¬ 
ing  environment  similar  or  identical  to  the  environment  in  which  per¬ 
formance  on  the  task  will  be  evaluated/measured.  In  the  area  of  flying 
training,  this  compelling  notion  has  led  to  an  unceasing  search  for  a 
learning  environment  for  student  pilots  as  much  like  the  performance 
environment  as  state-of-the-art  flight  simulator  technology  can  provide. 
While  there  are  significant  economic  factors  associated  with  the  search 
for  better  flight  simulator  fidelity,  as  the  business  of  making  a  simu¬ 
lator  as  much  like  the  aircraft  as  possible  has  come  to  be  known,  this 
discussion  will  not  deal  with  simulator  costs  directly.  Simulator  costs 
will  be  considered  peripherally,  later  in  the  presentation.  Instead, 
the  focus  will  be  on  flying  training  effectiveness,  or  on  the  question: 
Does  training  in  a  high  fidelity  simulator  transfer  to  an  aircraft  more 
effectively  than  training  in  a  low  fidelity  simulator? 

The  facts  to  be  presented  bearing  on  this  question  will  deal  with 
a  variety  of  flight  maneuvers,  learned  in  simulators  of  medium  and  high 
performance  aircraft,  and  tested  in  medium  and  high  performance  aircraft. 
The  facts  were  generated  by  both  student  and  instructor  pilots.  With 
three  exceptions,  the  data  represent  results  of  transfer  of  learning 
tests.  In  these  exceptions,  all  data  were  collected  in  the  simulator. 

The  specific  problems  attacked  were  the  training  effectiveness  of  a 
state-of-the-art  simulator  platform  motion  system  and  three  off-the- 
shelf  visual  display  systems  and  the  effects  of  simulator  motion  and 
visual  system  cueing  on  in-simulator  pilot  performance. 


I.  Transfer  of  Learning  Studies 

The  training  effectiveness  of  a  state-of-the-art  simulator  platform 
motion  system  was  evaluated  in  four  different  transfer  of  learning  stud¬ 
ies.  In  these  studies,  the  Advanced  Simulator  for  Pilot  Training  (ASPT) 
was  used  for  the  simulator  pretraining.  A  description  of  the  operational 
characteristics  of  the  ASPT  may  be  found  in  Gum,  Albery,  and  Basinger 
(1975) .  The  ASPT  is  equipped  with  a  six-post  synergistic  motion  system. 
In  all  of  these  studies,  the  motion  conditions  tested  were  full  platform 
motion  and  no  platform  motion.  The  wide  f ield-of-view  visual  display 
system  of  the  ASPT  was  fully  operational  in  each  case. 


The  first  of  the  studies  (Woodruff  et  al.,  1976)  employed  16 
Undergraduate  Pilot  Training  (UPT)  student  pilots,  four  of  whom  were 
trained  with  the  full  platform  motion  system  operative  while  another 
four  subjects  were  trained  in  the  ASPT  with  no  platform  motion.  The 
remaining  eight  subjects  received  no  ASPT  training  and  served  as  a 
control  group.  Simulator  pretraining  covered  all  major  areas  of  basic 
T-37  pilot  training.  Following  each  block  of  ASPT  pretraining,  all 
subjects  were  trained  to  Air  Training  Command  standard  criteria.  De¬ 
pendent  variables  were  T-37  training  hours  required  to  reach  criterion 
and  check  ride  scores.  Analyses  of  the  data  revealed  that  ASPT  trained 
subjects  required  fewer  hours  of  T-37  training  and  achieved  check  ride 
scores  that  were  equal  to  or  better  than  those  of  the  control  group. 

In  addition,  it  was  found  that  there  were  no  significant  or  practical 
differences  between  performances  of  students  trained  under  the  full 
platform  motion  and  the  no  motion  conditions.  The  main  results  of 
these  efforts  are  shown  in  Table  1. 

In  two  subsequent  studies,  the  contribution  of  a  synergistic  six 
degree-of-motion  platform  motion  system  to  the  acquisition  of  basic  con¬ 
tact  and  approach  and  landing  skills  (Study  I)  and  aerobatic  skills 
(Study  II)  was  reported  by  Martin  and  Waag  (1978a)  and  Martin  and  Waag 
(1978b)  .  Both  experiments  used  36  UPT  students  assigned  randomly  to 
three  treatment  groups:  (1)  ASPT  platform  motion  on;  (2)  ASPT  platform 
motion  off;  and  (3)  no  ASPT  training  (control).  All  ASPT-trained  stu¬ 
dents  received  the  same  amount  of  training:  10  instructional  sessions 
in  Study  I,  all  given  prior  to  training  in  the  T-37,  and  five  instruc¬ 
tional  sessions  in  Study  II,  three  given  before  initial  training  in  the 
T-37  (Aileron  Roll,  Split  S,  Loop,  and  Lazy  8)  and  two  given  later  fol¬ 
lowing  three  T-37  sorties  (Immelman,  Barrel  Roll,  and  Cuban  8).  Per¬ 
formance  of  the  ASPT  trained  students  was  assessed  during  ASPT  training 
by  means  of  the  ASPT  automated  performance  measurement  system  (Study  I) 
and  by  IP  ratings  and  special  data  records  (Study  II).  Transfer  of 
learning  was  evaluated  in  the  T-37  during  the  first  and  fifth  sorties 
by  IP  ratings  (Study  I)  and  during  the  three  initial  T-37  aerobatic 
sorties  and  an  additional  four  aircraft  sorties  following  completion  of 
the  final  ASPT  training  sessions  (Study  II) . 

The  results  of  analyses  of  the  data  from  these  two  experiments 
were  virtually  identical.  There  was  learning  during  training  in  the 
ASPT  for  the  motion  and  no-motion  trained  groups.  There  was  significant 
transfer  of  learning  to  the  T-37  for  the  ASPT  trained  students.  The 
transfer  effect  was  larger  in  Study  I  than  in  Study  II.  There  were  no 
differences  in  simulator  training  or  in  transfer  of  learning  to  the 
T-37  between  motion  and  no-motion  groups. 

Another  experiment  evaluated  the  contribution  of  ASPT  platform 
motion  to  air-to-surface  weapon  delivery  training  (Gray  and  Fuller, 

1977) .  In  this  study,  24  recent  UPT  graduates  were  assigned  equally  to 
three  groups.  Two  groups  were  trained  in  the  ASPT,  under  conditions 
of  full  platform  motion  and  no  platform  motion  while  the  third  group 
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received  no  ASPT  training.  The  ASPT  groups  were  trained  in  10,  15, 
and  20  degree  weapon  delivery  maneuvers.  Following  ASPT  training,  the 
motion  and  no  motion  group  subjects  dropped  bombs  using  the  F-5  air¬ 
craft.  Control  subjects  also  performed  the  weapon  delivery  maneuvers 
in  the  F-5.  Dependent  measures  were  bomb  miss  distances  and  number  of 
qualifying  bombs  dropped.  Analysis  of  the  data  again  showed  no  platform 
motion  effect  and  significant  transfer  of  learning  from  ASPT  to  the  air¬ 
craft.  It  is  worth  noting  that  these  subjects  participated  in  the 
study  following  substantial  T-38  training  before  and  after  completing 
UPT,  were  trained  in  the  ASPT,  a  T-37  simulator,  and  were  tested  for 
transfer  of  learning  in  the  F-5,  a  fighter  which  resembles  the  T-38 
but  in  which  none  of  the  subjects  had  any  previous  flying  experience. 

The  main  results  of  the  experiment  are  presented  in  Tables  2  and  3. 


Table  2 

Average  Bomb  Delivery  Circular  Error 


10°  dive 

15°  dive 

30°  dive 

ASPT-NM 

138* 

144' 

159’ 

ASPT— M 

148' 

138' 

169* 

Control 

200' 

180’ 

204' 

Table  3 

Qualifying  Bombs 


Number 

Percentage 

ASPT-NM 

42 

46 

ASPT— M 

41 

43 

Control 

24 

27 

The  results  of  these  four  experiments  indicate  that  simulator  plat¬ 
form  motion,  which  one  would  expect  to  increase  the  fidelity  of  the  ASPT, 
had  no  demonstrated  impact  on  the  training  effectiveness  of  the  simulator. 
Although  there  have  been  criticisms  of  the  operating  characteristics  of 


the  ASPT  platform  motion  system,  and  there  are  well  recognized  systematic 
problems  with  small  sample  studies  and  with  the  high  individual  perfor¬ 
mance  variability  characteristic  of  relatively  inexperienced  pilots, 
these  experiments  can  be  seen  as  a  basis  for  reevaluating  the  previously 
assumed  relationship  between  simulator  fidelity  and  training  effective¬ 
ness.  For  the  conditions  studied  and  the  maneuvers  trained,  it  would 
appear  that  the  lower  fidelity  simulation  was  as  good  as  the  higher  fi¬ 
delity  simulation  and  that  the  fidelity  principle  could  be  profitably 
reexamined,  as  far  as  platform  motion  systems  are  concerned. 

Moving  on  now  from  studies  of  the  effectiveness  of  simulator  plat¬ 
form  motion  systems,  we  can  review  an  experimental  evaluation  of  the 
training  effectiveness  of  three  different  typos  of  state-of-the-art  simu¬ 
lator  visual  systems  (Thorpe  et  al.,  1978).  In  this  test  of  simulator 
visual  system  training  effectiveness,  30  UPT  graduates  assigned  to  KC-135 
Combat  Crew  Training  (CCT)  were  randomly  assigned  to  three  groups  for 
simulator  pretraining  using  either  a  TV  model  board,  a  daylight  color  com¬ 
puter  generated  or  a  night-only,  point  light  source  color  computer  gener¬ 
ated  visual  display.  All  students  were  given  eight  1-hour  simulator 
training  sessions  focused  on  the  turn  to  final  approach  and  landing. 

The  transfer  of  learning  test  was  accomplished  in  the  KC-135  immediately 
following  completion  of  simulator  training.  Additional  transfer  data 
were  obtained  from  final  checkride  evaluations  at  the  end  of  CCT  for  the 
30  students  in  the  experiment  compared  with  the  final  checkride  evalua¬ 
tions  of  a  number  of  previous  CCT  classes.  The  dependent  measures  were 
IP  ratings  based  on  objective  behavioral  criteria. 

Analyses  of  the  data  revealed  that  there  are  no  differences  among 
the  three  visual  systems  in  terms  of  the  in-simulator  performance  of  the 
students.  In  the  KC-135  transfer  test,  students  trained  using  the  day 
color  and  night  only  computer  generated  displays  performed  bettor  than 
the  TV  model  board-trained  students.  These  differences  appeared  during 
the  final  approach  and  landing  portions  of  the  training  task.  In  the 
final  CCT  evaluations,  the  day  color  group  performed  better  than  the 
night-only  and  TV  groups,  and  the  simulator-trained  students  of  this 
study  received  higher  ratings  than  previous  CCT  classes  which  had  not 
received  similar  simulator  pretraining. 

The  results  of  this  evaluation  of  the  training  effectiveness  of 
different  simulator  visual  systems  are  interpreted  as  supporting  the 
outcome  of  the  transfer  of  learning  tests  of  simulator  platform  motion 
reported  above.  In  this  case  the  computer  generated  displays  were  more 
effective  than  the  TV-model  system.  The  computer  generated  displays 
were  less  graphically  realistic;  that  is,  lower  fidelity  than  the  TV- 
model  system.  They  were  like  cartoons  of  the  visual  environment  as 
contrasted  with  a  television  image  of  a  similar  visual  environment. 

While  the  result  of  the  simulator  visual  system  training  effectiveness 
study  is  viewed  here  as  showing  that  the  less  realistic,  lower  fidelity 
system  was  more  effective,  it  should  be  noted  that  the  hardware  systems 
used  were  off-the-shelf  simulator  visual  displays  which  were  not  matched 


49 


in  any  systematic  way  on  such  features  as  resolution,  field  of  view, 
or  brightness  but  were  used  as  intact  units  supplied  under  contract 
for  this  research  effort. 

To  summarize  this  review  of  five  transfer  of  learning  studies  of 
simulator  training  effectiveness,  it  becomes  clear  that  simulator  pre¬ 
training  is  effective,  and  that  the  differences  in  training  effective¬ 
ness  as  a  function  of  simulator  realism/fidelity  measured  in  these 
studies  are  relatively  small  contrasted  with  the  transfer  effects  they 
generate . 


II.  Simulator  Effectiveness  Evaluations 


This  section  will  describe  three  studies  in  which  the  effective¬ 
ness  of  simulator  design  characteristics  were  measured  in  terms  of  the 
in-simulator  performance  of  experienced  pilots.  The  first  two  of  these 
studies  (Irish  et  al.,  1977,  and  Irish  &  Buckland,  1978)  investigated 
a  relatively  large  number  of  hardware  variables  first  in  less,  and  then 
more  conservative  adaptations  of  efficient,  multifactor  research  designs 
(Simon,  1973).  These  studies  were  accomplished  in  the  ASPT  and  varied 
combinations  of:  (1)  Platform  motion  conditions  (no  motion,  three 
degrees-of-freedom  (DOF)  and  six  DOF)  ;  (2)  G-seat,  a  motion  cueing  de¬ 
vice  which  simulates  sustained  G-forces  by  inflating  and  deflating 
bladders  in  the  pilot's  seat  in  accordance  with  a  pilot's  aircraft  con¬ 
trol  inputs  (no  G-seat,  seat  pan  only,  and  G-seat  fully  operational)  ; 
and  (3)  visual  display  field  of  view  (full  FOV  and  36°V  x  4B°h,  Study  I, 
and  full  FOV,  36«>V  x  48°H,  and  36°V  x  144°H,  Study  II).  Other  variables 
were  manipulated  in  these  experiments  but  we  will  focus  on  the  effects 
of  platform  motion,  G-seat,  and  field-of-view  variations  on  the  in¬ 
simulator  performance  of  eight  experienced  UPT  IP's  (three  in  Study  I, 
five  in  Study  II)  who  flew  repeated  takeoffs,  overhead  traffic  patterns. 
Ground  Controlled  Approaches  (GCA's),  aileron  rolls,  and  slow  flight 
exercises  (Study  I)  and  overhead  traffic  patterns,  GCA's,  loops,  and 
barrel  rolls  (Study  II) . 

The  results  of  the  two  studies  were  consistent.  Performance  was 
significantly  better  under  wide  than  narrow  FOV  conditions.  Performance 
was  significantly  better  under  no  platform  motion  than  under  three  or 
six  DOF  conditions  and  there  was  no  significant  G-seat  effect  on 
performance. 

These  studies  can  be  interpreted  as  suggesting  that  larger  sized, 
cartoon-style  visual  displays,  which  may  be  more  realistic  than  smaller 
sized,  cartoon-style  displays,  lead  to  improved  pilot  performance  and 
therefore  support  the  realism/fidelity  principle.  On  the  other  hand, 
the  information  is  not  really  more  realistic/higher  fidelity;  there  is 
simply  more  of  it  presented  simultaneously.  Call  this  a  draw,  as  far 
as  acceptance/rejection  of  the  fidelity  hypothesis  goes. 
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In  the  case  of  tho  platform  motion  and  G-seat  variables,  the  pat¬ 
tern  encountered  with  relatively  inexperienced  pilot  students  reapi>ears. 
That  is,  where  simulated  cockpit  movement  and  simulated  ('.-forces  are 
present,  their  absence  in  a  simulator  must  be  reyarded  as  lower  fidelity 
and  the  observed  significantly  better  performance  under  the  conditions 
of  no  platform  motion  and  a  nonoperational  G-seat  must  indicate  the  in¬ 
effectiveness  of  those  hardware  features  and  therefore  the  lack  of  ef¬ 
fectiveness  of  higher  as  compared  with  lower  fidelity. 

On  tho  other  hand,  it  must  be  noted  that  the  results  were  the  in¬ 
simulator  performances  of  experienced  pilots  who  are  widely  known  to 
adapt  rapidly  so  as  to  make  the  best  of  any  aircraft  control  problem. 

So  again,  at  best,  there  is  only  the  additional  suggestion  that  recon¬ 
sideration  of  the  validity  of  the  assumption  that  simulator  realism/ 
fidelity  produces  training  effectiveness  may  be  warranted. 

The  final  study  bearing  on  the  fidelity  issue  to  be  summarised  in 
this  presentation  involves  the  use  of  the  ASPT  in  a  study  of  the  effects 
of  an  Area-of-Interest  display  FOV  size  on  the  accuracy  of  air-to-surface 
weapon  delivery  performance  of  eight  experienced  tactical  fighter  pilots 
(LcMastcr  &  Longridge,  1978). 

In  this  study  there  wore  other  variables  but  again  we  will  focus 
only  on  the  effects  of  the  Area-of-Interest  display  FOV  size.  There 
were  five  different  FOV  sizes  tested:  (1)  38°V  x  52°H;  (.')  70°V  x  70°Hj 
(3)  70°V  x  90°H,*  (4)  70°V  x  110°M;  and  (5)  70OV  x  130°H.  The  results 
show  that  bombing  accuracy  is  siqnificantly  better  when  the  three  larger 
FOV  sizes  wore  used  and  that  the  bomb  delivery  scores  obtained  using 
these  three  FOV  conditions  are  not  significantly  different  from  each 
other.  The  results  are  shown  in  Figure  5-1.  The  results  are  interpreted 
as  a  determination  of  the  amount  of  roalism/fidelity  needed  for  a  spe¬ 
cific  task  and  represent  something  of  a  novelty  in  simulator  requirements 
resoarch,  a  reasonably  clear-cut  answer  to  a  real  problem  with  enough 
excess  information  to  apply  intelligently  to  answer  the  what- if  questions 
that  often  arise. 


III.  What  Does  It  All  Mean? 

Q.  Now  with  the  facts  before  us,  what  can  be  made  of  them? 

A.  We  seem  to  have  demonstrated  the  ineffectiveness  of  a  state-of- 
the-art  platform  motion  system  to  influence  the  in-simulator  or  trans¬ 
fer  of  learning  performance  of  UPT  students  in  a  variety  of  UPT  training 
tasks.  Further,  tho  platform  motion  was  shown  to  generate  degraded  in¬ 
simulator  performance  by  experienced  IP's  in  a  variety  of  UPT  maneuvers, 
and  no  consistent  G-seat  effect  was  found  on  the  performances  of  tho  IP's 
in  these  maneuvers. 
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Q.  What  else? 


A.  The  data  on  platform  motion  has  been  interpreted  as  suggesting 
that  lower  fidelity  simulators  may  be  as  effective  as  higher  fidelity 
simulators.  This  tentative  conclusion  is  supported  also,  but  less  em¬ 
phatically,  by  the  KC-135  visual  system  effort.  In  addition,  the  data 
illustrate  that  the  procedures  used,  namely  the  classical  transfer  of 
learning  paradigm  and  the  in-simulator  differential  objective  performance 
test  of  skilled  pilots,  seem  to  be  sensitive  enough  to  reveal  signifi¬ 
cant  differences  when  they  exist. 

Q.  So  what? 

A.  To  answer  this  question,  let  us  consider  practical  issues,  such 
as  reliability,  maintainability ,  and  cost-effectiveness.  It  may  be  that 
these  key  characteristics  of  flight  simulators  are  impacted  by  realism/ 
fidelity  requirements.  Typically,  higher  fidelity  means  higher  costs, 
lower  maintainability  and  reliability,  and,  at  least  in  some  cases,  as 
shown  above,  decreased  cost  effectiveness.  Such  considerations  are  not 
trivial. 


Q.  Finally,  what  shall  we  do  about  fidelity? 

A.  Probably  it  would  be  wise  to  regard  it  suspiciously  until  the 
research  reviewed  above  is  confirmed  or  discontinued  in  subsequent  studies 
using  the  ASPT  and  other  simulators  in  other  laboratories;  e.g.,  the 
Aviation  Wide  Angle  Visual  Simulator  of  the  Naval  Training  Equipment 
Center.  Meanwhile,  we  can  remind  ourselves  as  the  design  of  a  simulator 
is  being  determined  that  what  can  be  practiced  cam  be  learned,  what 
can  be  learned  can  be  transferred,  and,  as  soon  as  the  capabilities  of 
a  simulator  support  practice  of  a  set  of  training  tasks,  stop  adding  to 
it.  Everything  else  the  simulator  does  will  cost  more,  and  while  it  may 
not  detract  from  training  effectiveness,  it  probably  won't  add  to  it. 
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6.0  A  DISCUSSION  OF  PSYCHOLOGICAL  FIDELITY  IN  SIMULATED 

WORK  ENVIRONMENTS 

Earl  A.  Alluisi 

Performance  Assessment  Laboratory 
Old  Dominion  University 


Four  highly  relevant,  scholarly  papers  have  been  presented.  Col¬ 
lectively,  the  four  papers  constitute  an  excellent  state-of-the-art 
review  of  data,  methods,  and  theory  regarding  the  importance  of  func¬ 
tional  (behavioral  or  psychological)  fidelity  versus  physical  (engineer¬ 
ing  or  hardware)  fidelity  as  a  requirement  for  simulators,  or  for  situ¬ 
ations  in  which  some  degree  of  simulation  is  involved.  Yet,  the  question 
remains  unanswered,  for  the  review  indicates  that  the  data  are  insuffi¬ 
cient,  the  methods  inadequate,  and  the  theory  undeveloped — and  one  of 
the  valuable  outcomes  of  the  review  provided  in  this  symposium  lies, 
without  doubt,  in  the  report  of  progress  made  and  suggestions  for  direc¬ 
tions  to  be  followed  in  the  future. 

After  presenting  a  carefully  thought-out  definition  of  functional 
(which  he  calls  "behavioral")  fidelity  that  differentiates  it  from 
physical  fidelity,  W.  G.  Matheny  concentrates  on  the  tracking  (or 
"closed-loop  control")  aspects  of  flight  simulation.  His  thesis  is 
that  system  identification  methods  can  be  used  to  describe  the  operator’s 
behavior(s)  in  such  systems — e.g.,  one  involving  airborne  flight  and 
another  involving  ground-based  simulated  flight.  If  the  analysis  indi¬ 
cates  that  the  two  systems  are  behaviorally  equivalent,  it  can  be  ex¬ 
pected  (and  tested  empirically)  that  training  or  practice  in  either 
would  transfer  positively  to  the  other.  The  empirical  tests  are  called 
for  in  order  to  make  decisions  regarding  the  extent  to  which  various 
physical  differences  in  alternative  simulation  situations  reflect  them¬ 
selves  in  changes  in  the  operator's  behavior.  He  argues  that  the  analy¬ 
sis  (and  test)  for  behavioral  similarity  is  the  crucial  key  to  advance¬ 
ment  toward  the  answer  to  the  questions  regarding  the  importance  of 
different  degrees  or  kinds  of  fidelity  in  simulation.  He  rejects  the 
use  of  transfer-of-training  data  for  such  purposes,  arguing  that  too 
many  extraneous  variables  may  influence  the  results  to  an  unacceptable 
degree  in  such  studies.  He  is  pessimistic  in  evaluating  past  progress, 
stating  that  during  the  last  quarter  century  little  gain  has  been  made 
toward  answering  the  "behavioral  similarity  question."  He  is  optimistic 
about  the  future,  saying  that  use  of  the  system  identification  procedures 
to  study  the  behavioral  similarity  question  is  both  feasible  and  of  po¬ 
tentially  high  payoff. 

In  the  second  paper,  F.  T.  Eggemeier  and  B.  W.  Cream  consider  the 
question  of  fidelity  with  respect  to  simulators  used  for  crew,  group, 
team,  or  unit  (CGTU)  training.  They  note  that  the  currently  available 
techniques  of  task  analysis,  even  in  the  more  advanced  Instructional 
Systems  Development  (ISD)  or  Systems  Approach  to  Training  (SAT) 


methodologies,  are  inadequate  for  the  specification  of  CGTU  (as  con¬ 
trasted  with  individual)  training  requirements.  They  go  on  to  propose 
a  Criticality,  Frequency,  Difficulty  (CFD)  task-analytic  technique  to 
specify  the  design  of  training  devices  with  "only  the  levels  of  fidelity 
necessary"  to  accomplish  specific  CGTU  training  missions.  The  technique 
involves  the  training  psychologist  as  a  catalyst  in  bringing  together 
the  user's  content  information  and  the  simulation  engineer's  expertise. 
Their  reported  experience  with  the  Gunship  trainer  clearly  indicated 
the  need  to  consider  CGTU  training  requirements  in  the  IDS/SAT  process — 
they  found  substantial  implications  for  the  design  of  both  the  crew  and 
the  instructor  stations.  Their  evaluation  of  the  AC-130E  Gunship  train¬ 
ing  device  showed  that  the  trainees  (a)  reached  and  maintained  criterion 
performance  with  fewer  airborne  missions  (i.e.,  there  was  positive 
transfer  of  training) ,  and  (b)  were  either  not  different  or  better  in 
proficiency  than  controls  who  were  trained  exclusively  in  airborne  mis¬ 
sions  without  benefit  of  the  trainer.  (Remember,  Matheny  would  probably 
dispute  the  importance,  if  not  the  validity,  of  the  interpretation  made 
on  the  basis  of  any  transfer-of- training  study.)  The  authors  also  de¬ 
fine,  and  distinguish  among,  (a)  man-ascendant  versus  machine-ascendant 
systems  on  the  basis  of  the  proportion  of  system  performance  that  is 
determined  by  interpersonal  interactions  among  crewmembers,  (b)  S-R 
versus  organismic  models  of  CGTU  behavior  on  the  basis  of  the  degree 
to  which  contextual  or  Gestalt-like  properties  do  not  influence  out¬ 
comes  in  the  model,  and  (c)  the  different  task  demands  of  established 
versus  emergent  situations  on  the  basis  of  the  extent  to  which  desired 
actions  (or  a  "school  solution")  can  be  specified.  They  note  that  the 
technique  that  they  propose  works  best  (a)  with  S-R  analysis,  (b)  in 
established  situations,  (c)  of  either  or  both  man-  and  machine-ascendant 
systems;  its  applicability  in  emergent  situations  has  yet  to  be 
demonstrated. 

For  his  paper,  E.  E.  Eddowes  adopts  the  view  that  the  "goodness" 
or  "desirability"  of  a  given  kind  or  amount  of  fidelity  in  a  simulator 
should  be  judged  in  terms  of  the  effectiveness  of  the  device  for  train¬ 
ing.  He  then  reviews  the  results  of  five  rather  impressive  studies  of 
the  use  of  flight  simulation  for  military  training.  The  findings  clearly 
appear  to  be  that  pre-airborne  flight  experience  in  the  way  of  simulator 
pretraining  is  effective  for  training  in  the  sense  that  it  reduces  the 
requirements  for  airborne-flight  training  time.  Also,  within  a  surpris¬ 
ingly  wide  range  of  differences  in  platform  motion  and  visual  systems, 
the  differences  in  simulation  training  effectiveness  as  a  function  of 
the  realism  or  physical  fidelity  was  surprisingly  small  relative  to  the 
large  positive  transfer  effects  produced  by  use  of  simulation  in  the 
first  place!  The  practical  implications  of  these  findings- -in  terms  of 
such  considerations  as  the  reliability,  maintainability,  and  cost- 
effectiveness  of  flight  simulators  of  different  design — arr  quite  im¬ 
portant.  And  in  spite  of  his  adoption  of  a  transfer-of-training  cri¬ 
terion  for  evaluating  the  "goodness"  of  a  simulator's  fidelity,  Eddowes 
approaches  Matheny 's  position  near  the  end  of  his  paper  when  he  says, 

".  .  .  what  can  be  practiced  can  be  learned,  what  can  be  learned  can 
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be  transferred,  .  .  Matheny's  point  earlier  having  been  that  one 
needs  to  know  what  has  to  be  transferred  in  order  to  know  what  should 
be  practiced  and  learned.  Data  such  as  those  presented  by  Eddowes  will 
be  necessary,  however,  to  permit  the  development  of  meaningful  theory 
such  as  that  called  for  by  Matheny,  and  perhaps  there  is  real  justifi¬ 
cation  for  optimism  about  the  future! 

Such  optimism  is  further  justified  by  the  material  reviewed  and 
presented  by  D.  E.  Erwin  in  the  fourth  and  final  pa^  -»r.  The  quality 
or  "goodness"  of  functional  (behavioral  or  psychological)  fidelity  of 
a  simulator  or  simulation  situation  is  defined  in  terns  of  the  degree 
of  positive  transfer  obtained  (or,  inversely,  in  terms  of  the  degree 
to  which  the  transfer-of-training  discrepancy  is  minimized) .  Functional 
fidelity  is  distinguished  from  physical  (or  engineering)  fidelity;  they 
are  said  to  represent  different  (but,  perhaps,  sometimes  correlated) 
dimensions,  and  the  point  is  made  that  fully  engineered  fidelity  may 
not  be  optimum  in  a  simulator  for  training.  The  author  then  describes 
the  approach  being  taken  at  the  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  in  developing  engagement  simulation 
training  systems — for  use,  of  course,  in  collective  training  of  CGTU's 
in  man-ascendant  tactical  environments.  He  points  out  that  the  simula¬ 
tion  must  be  to  a  considerable  extent  functional,  since  the-physical 
fidelity  of  combat  simulation  is  not  feasible.  The  system  simulated 
is  man-ascendant,  and  the  situation  is  emergent  rathep  than  established; 
the  organismic  model  is  adopted  as  seemingly  much  more  representative 
of  CGTU  behavior  than  the  S-R  model.  Erwin  agrees  with  the  point  made 
previously  by  Eggemeier  and  Cream  that  the  currently  available  task 
analytic  ISD/SAT  techniques  appear  to  have  little  applicability  in 
these  circumstances.  What  is  promised,  however,  is  the  development, 
from  engagement-simulation  exercises  of  a  data  base  of  training -product 
measures  (such  as  battle  wins  and  casualty-exchange  ratios)  as  well  as 
potential  CGTU-training-process  measures  (such  as  communication  and 
movement  patterns)  ,  with  the  possibility  of  correlating  the  two.  The 
empirically  valid  development  of  such  criteria  would  provide  a  major 
breakthrough  for  CGTU  military  training  R&D;  the  feasibility  of  such 
a  development  is  a  major  source  of  the  optimism  cited. 


Some  Additional  Views 


A  discussant  has  the  responsibility  not  merely  to  summarize  or 
evaluate  the  major  contributions  of  the  presentations  in  a  symposium, 
but  also  to  add  whatever  additional  views  are  judged  appropriate  to 
balance  the  presentation  and  to  emphasize  the  state-of-affairs  that 
seems  to  follow  from  an  overview.  In  attempting  to  fulfill  that  re¬ 
sponsibility,  I  shall  slip  into  the  first  person;  this  should  serve 
as  a  reminder  that  the  expertise  lies  with  the  other  members  of  the 
symposium — I  am  only  a  poor  "outsider"  looking  ini 
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Extents  of  Physical  and  Functional  Fidelity.  As  one  of  the  pre¬ 
vious  speakers  noted,  a  flight  simulator  does  not  move  like  the  air¬ 
craft  it  simulates.  When  we  speak  of  "physical  fidelity"  in  a  simulator, 
we  are  not  speaking  of  a  categorical  characteristic  that  the  simulator 
either  has  or  fails  to  have.  Rather,  physical  fidelity  exists  in  some 
degree  in  every  simulator,  but  to  different  degrees  in  different  simu¬ 
lators.  An  underlying  concern  of  the  topic  of  this  symposium  is  the 
question  of  the  optimum  degree  of  physical  fidelity.  Underlying,  but 
often  unstated  assumptions  are  that  increasing  degrees  of  physical 
fidelity  (certainly  as  equivalence  to  the  system  simulated  is  approached) 
are  associated  with  (a)  highei  costs,  and  (b)  increased  effectiveness. 
Both  assumptions  are  capable  of  being  tested  empirically,  and  indeed 
the  extent  to  which  either  is  valid  could  be  demonstrated  by  determina¬ 
tion  of  the  functional  relation  (graph  or  equation)  of  (a)  cost  or 
(b)  effectiveness  as  a  function  of  degree  of  physical  fidelity.  With 
few  exceptions,  this  has  not  been  attempted  for  any  of  the  simulators 
with  which  I  am  familiar. 

On  the  other  hand,  we  also  speak  of  "functional  (behavioral  or 
psychological)  fidelity."  This,  too,  should  be  thought  of  as  a  contin¬ 
uum,  rather  than  as  a  categorical  characteristic  of  a  simulator.  The 
express  concern  of  this  symposium  has  to  do  with  the  optimum  degree  of 
functional  fidelity.  Underlying  assumptions  here,  also  often  left 
unstated,  are  that  relative  to  equal  or  lesser  degrees  of  physical 
fidelity,  functional  fidelity  (a)  costs  less  for  equal  effectiveness, 
or  (b)  provides  greater  effectiveness  at  equal  or  lesser  cost.  As  was 
the  case  with  physical  fidelity,  these  assumptions  regarding  functional 
fidelity  are  capable  of  being  tested  empirically.  To  some  extent  they 
have  been,  and  are  being,  tested  as  indicated  by  the  material  presented 
in  the  papers  of  the  four  principal  contributors  to  this  symposium. 

Although  physical  and  functional  fidelities  can,  and  even  perhaps 
should,  be  viewed  as  separate  dimensions,  they  are  never  entirely  un¬ 
related;  rather,  they  are  something  like  oblique  factors — they  are  cor¬ 
related,  but  at  best  only  moderately  so.  This  being  the  case,  there 
should  be  no  surprise  at  finding  "an  engineering  bias"  in  the  direction 
of  producing  a  simulator  the  static  and  dynamic  characteristics  of  which 
most  closely  resemble  those  of  the  system  simulated;  i.e.,  a  bias  toward 
trying  to  design  and  develop  and  produce  simulators  with  high  degrees  of 
physical  (or  engineering  or  hardware)  fidelity. 

Likewise,  there  should  be  no  surprise  at  finding  "a  psychological 
bias"  in  the  direction  of  producing  a  simulator  that  stimulates  in  the 
human  operator  those  static  and  dynamic  responses  which  most  closely 
resemble  those  of  the  same  human  operator  in  the  system  simulated;  i.e., 
a  bias  toward  trying  to  design  and  develop  and  produce  simulators  with 
high  degrees  of  functional  (or  behavioral  or  psychological)  fidelity. 
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Simulators  for  Military  Training.  Such  biases  and  points-of- 
view  are  seldom  amenable  to  direct  empirical  verification.  However, 
outcomes  based  on  their  application  can  be  tested  and  inferences  then 
made  regarding  the  likely  general  validity  of  the  biases  themselves. 

That  is  to  say,  in  practical  problem  solving  situations,  different  bi¬ 
ases  generally  lead  to  proposals  of  alternative  solutions,  and  the  so¬ 
lution  selected  as  "best"  serves  indirectly  to  verify  the  bias  out  of 
which  it  was  derived.  The  selection  of  one  of  a  number  of  possible 
solutions  as  "best"  in  a  practical  situation  generally  requires  a  cri¬ 
terion,  and  a  practical  criterion  generally  implies  some  agreement  on 
the  area  of  use  or  utility  of  the  item  or  solution  selected. 

In  order  to  make  these  generalizations  more  concrete,  let  us  con¬ 
sider  the  use  of  simulators  in  the  Military  Departments.  For  the  most 
part,  they  are  used  for  training  or  for  exercising— i.e. ,  for  aiding  in 
the  acquisition  of  skill  by  essentially  unskilled  individuals  or  CGTU's, 
as  in  the  use  of  flight  simulators  in  undergraduate  pilot  training 
(UPT)  ,  or  for  aiding  in  the  maintenance  of  proficiency  by  already  skilled 
individuals,  as  in  the  use  of  flight  simulators  by  pilots  to  maintain 
their  piloting  skills  when  for  whatever  reason  the  amount  of  actual  fly¬ 
ing  time  is  deemed  insufficient.  Both  uses  require  appropriate  prac¬ 
tice,  with  positive  transfer,  but  there  is  no  reason  to  assume  a  priori 
that  what  is  an  optimum  degree  of  functional  (or  physical)  fidelity  for 
the  one  use  will  be  the  same  as  that  for  the  other  use.  For  the  pur¬ 
poses  of  further  discussion,  I  shall  restrict  myself  to  the  use  of  simu¬ 
lators  for  military  training.  And  in  this  regard,  it  might  be  well  to 
think  through  once  again  why  we  are  in  the  training-simulator  business 
at  all! 

We  start  with  a  requirement — in  this  case  a  job  we  want  a  person 
to  do  or,  from  another  point  of  view,  a  person  who  we  want  to  be  able 
to  do  a  certain  job.  In  former  days,  centuries  ago,  all  craftsmen 
learned  their  trade  through  an  apprenticeship  method;  even  fighting 
men  progressed  to  knighthood  through  an  apprenticeship  system.  Years, 
and  sometimes  decades,  separated  the  stages  of  skill  denoted  by  "ap¬ 
prentice,"  " journeyman, "  and  "master."  The  system  worked,  but  it  was 
relatively  inefficient  and  inflexible — especially  when  faced  with  the 
problem  of  increasing  quickly  the  numbers  of  available  journeymen  and 
masters.  So  we  then  developed  schools  and  training  centers  to  provide 
for  increased  efficiency  and  flexibility,  especially  for  "apprentice¬ 
ship"  training  such  as  UPT.  Once  we  developed  training  centers,  we 
had  persons  who  began  to  interest  themselves  in  the  quality  and  effi¬ 
ciency  of  the  training  techniques  employed.  This,  in  turn,  has  led  us 
from  an  "apprentice-like"  system  of  training,  where  for  example  an 
instructor  pilot  trains  a  student  pilot  in  UPT  with  an  aircraft  and  no 
other  training  devices,  toward  a  more  efficient  system  of  training  that 
employs  equipment,  simulators,  and  training  devices  in  addition  to  in¬ 
structor  pilots  and  aircraft.  Now  necessity,  that  mother  of  so  much 
invention,  is  encouraging  us  to  move  further — to  specify  in  concrete 
terms  the  relative  merits  of  alternative  training  devices,  equipments, 
simulators,  and  systems.  And  the  concrete  terms  demanded  are  those  of 
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the  cost-effectiveness  of  the  alternatives  for  the  purposes  proposed— 
e.g.,  the  cost-effectiveness  of  flight  simulators  for  military  train¬ 
ing,  a  topic  recently  addressed  by  Orlansky  and  String  (1977)  in  a  two- 
volume  report. 

It  seems  to  me  that  the  question  of  the  optimum  degree  (s)  of 
functional  (and  physical)  fidelity  in  a  simulator  for  military  train¬ 
ing  can  and  should  be  reduced  to  the  question  of  the  cost-effectiveness 
of  the  alternatives  when  used  for  specific  and  appropriate  training  pur¬ 
poses.  Of  course,  that  requires  both  the  analysis  of  the  cost  factors, 
and  the  measurement  of  the  effectiveness-for-training  factors,  for  all 
alternatives — and  for  different  methods  of  use.  But  these  are  empiri¬ 
cal,  not  philosophical,  questions,  and  although  they  are  difficult, 
they  are  not  impossible  to  answer. 

(As  an  aside,  this  same  approach  should  be  applied  at  a  higher 
level,  for  example  to  aid  in  making  decisions  regarding  alternative 
solutions  to  the  problem  of  meeting  training  requirements.  In  the  train¬ 
ing  domain,  we  are  and  should  be  working  to  find  ways  of  training  people 
to  behave  in  desired  ways  and  to  do  so  more  effectively  with  less,  and 
less  costly,  training;  in  the  human-factors  engineering  domain,  we  are 
and  should  be  working  to  design  systems  in  such  ways  as  to  require 
less  skilled  persons  to  operate  them,  thereby  reducing  training  re¬ 
quirements  and  minimizing  the  effects  of  behavioral  variations  on  sys¬ 
tem  performance.  The  evaluation  of  the  training  versus  alternative- 
design-of-hardware  alternatives  should  be  on  the  basis  of  their  relative 
cost-effectiveness  ratios.) 

Simulators  and  CGTU  Training.  When  we  move  from  individual  to 
CGTU  training,  the  problems  (and  our  data  needs)  seem  to  increase  by 
orders  of  magnitude.  Perhaps  because  of  this,  until  recently  there 
has  been  relatively  little  R&D  on  CGTU  training  in  the  Military  Depart¬ 
ments— this  in  spite  of  the  fact  that  the  vast  majority  of  Service 
training  and  exercising  was  estimated  to  be  in  the  domain  of  collective 
training  in  units.  The  Army  studies  reported  by  Erwin,  and  the  Air 
Force  studies  reported  by  Eggemeier  and  Cream  are  welcome  indications 
of  R&D  efforts  and  progress  in  this  difficult  area — an  area  necessarily 
characterized  at  this  stage  with  organismic  models  of  man-ascendant 
systems  operating  in  emergent  situations,  to  which  our  previously  de¬ 
veloped  task-analytic  ISD/SAT  tools  seem  ineffective.  Yet,  progress 
is  clearly  being  made,  and  the  promise  for  the  future,  e.g.,  in  the 
development  of  both  product  and  process  criteria  of  CGTU  training  in 
the  Army,  is  quite  great. 

The  need  for  attention  to  R&D  on  CGTU  training  has  been  recognized 
and  published.  The  Defense  Science  Board  Task  Force  on  Training  Tech¬ 
nology  in  1976  called  this  "an  area  in  which  significant  improvements 
in  efficiency  and  effectiveness  are  now  possible"  and  recommended  in¬ 
creases  in  technology-base  funds  for  training  technology  R&D  in  support 
of  crew,  group,  team,  and  unit  (CGTU)  training  (Alluisi,  1976,  p.  xi) . 
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During  that  same  year,  the  Defense  Science  Board  Task  Force  on  Tech¬ 
nology  Base  Strategy  listed  "Training  RfiD"  as  the  first  of  nine  areas 
in  which  increased  technology-base  investment  seems  warranted,  com¬ 
menting  that  the  development  of  CGTU  training  "shows  great  promise" 
(Rasmussen,  1976,  pp.  4  £  15) .  In  his  paper  today,  Erwin  documented 
John  Allen's  support  and  extension  of  these  positions  while  he  served 
as  Deputy  Director  of  Defense  Research  and  Engineering  (Research  and 
Advanced  Technology).  I  shall  not  quote  him  again  here,  but  say  merely 
that  he  expressed  recognition  of  the  need  for  increasing  R&D  on  CGTU 
training,  on  the  use  of  simulators  in  training,  and  on  the  two  topics 
together  (cf .  Allen,  1977a,  b) . 

The  interests  of  science  can  be  well  served  by  the  adoption  of 
a  cost-effectiveness  criterion  approach  to  the  topic  of  this  symposium 
and  training  R£D  generally. 
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